Translate

Archives

The Future of the Korn Shell may be in Dangerous Hands

Until his employment was terminated by AT&T Research Labs, David (Dave) Korn was the primary maintainer/gatekeeper who controlled any changes to the ksh93 source code base. Over the years, this resulted in a series of very stable releases with excellent backward compatibility across a wide range of platforms.

On October 1st 2013, Glenn Fowler, who was the primary maintainer of many of the AST (AT&T Software Technology) libraries used by ksh93, posted this to the ast-users mailing list:

As has been pointed out several times on the AST and UWIN lists, AT&T gives very little support to OpenSouce software, which is why we have so few people involved with our rather large collection of AST software. In spite of this, ksh, nmake, vczip, UWIN and other AST tools continue to be used in several AT&T projects.

It turns out that software isn’t the only thing lacking support: both dgk (David Korn) (AT&T fellow, 36 years of service) and gsf (Glenn Fowler) (AT&T fellow, 29 years of service) have been terminated, effective October 10. Our third major partner, Phong Vo (AT&T fellow, 32 years of service), left a few months ago for Google. The UWIN maintainer, Jeff Fellin, is still with AT&T and provides UWIN support for some critical operations.

Both dgk and gsf will continue to work on AST software, and might actually have more time (at least in the short run) to focus on it.

The download site and mail groups will remain within AT&T for at least the next several months. Our AT&T colleague, dr.ek, AST user and bug detector, will maintain the site. We have secured the astopen.org domain and are investigating non-AT&T hosting options, including a repository with bug tracking.

The process of change will take time; the patience of the user community will be greatly appreciated. Its quite a shock to have 3 weeks to plan personal, career, and hacking futures after working in an environment that has essentially been stable for almost 30 years. The user groups will be informed as plans solidify.

David and Glenn were hired by Google within a few months of being layed off and no longer participate in maintaining or enhancing ksh93 or the underlying ast libraries.

In early 2016, Eleftherios Koutsofios of AT&T Labs posted the following message to the AST and UWIN mailing lists:

hi AST and UWIN users.

as many of you noticed, the download site on www.research.att.com went off the air shortly before the end of the year due to some security issue.

the timing was unfortunate because several people including me were on vacation so it’s been down for a long time.

but we’ve finally managed to move most of that software on GitHub. you can find the AST and UWIN software packages at:

https://github.com/att/uwin and https://github.com/att/ast

(btw. the /att tree on GitHub hosts a lot of open source software developed by the AT&T Research group. feel free to browse. I’ll be putting up some of my code there soon).

/att/ast corresponds to the ast-open package. it includes the software that was also available under individual packages, like ast-ksh, ast-dss, etc., so I decided to only create this one. it has 3 branches, matching the old structure: master (i.e. official), alpha, and beta. beta is the most recent one. it includes the last package I had gotten from Glenn and Dave with some minor fixes to get it to compile on some new OS versions, like Centos 7 and Ubuntu 14.

/att/uwin is the source code for the UWIN system. it has a master and a beta branch. I don’t have an environment to build and test this on, so I don’t know how well it builds.

cloning either of these git repos is equivalent to downloading the INIT and ast-open (or INIT and uwin) packages from the old site and then running:

./bin/package read
so the next step after the clone step is to run:

./bin/package make
vanilla build, where no previous version of NMAKE is available should still work and on some systems that was actually the way to go for me.

as an example, to get and compile the beta branch of AST:

git clone –branch beta \
https://github.com/att/ast.git
cd ast
./bin/package make
very little of the documentation from the old site has moved to the GitHub site, I’ll try to migrate the rest later, I just wanted to get the software up again.

thanks lefteris

In June 2017, a Red Hat employee, Siteshwar Vashisht, based in Brno, Czech Republic, announced to whoever reads his blog that “KornShell is not dead”.

Despite the lack of pace in development from last couple of years, ksh still remains one of the most popular shells. AT&T released source code on GitHub, but there were no updates from them for more than an year. Last commit was made by AT&T on Jan 11, 2016. I have been submitting bug fixes on the mailing list, but the patches were not added to the upstream repository as there is nobody working on KornShell at AT&T. This week they provided me commit access to the GitHub repository. The latest code was released under beta branch and I am going to build on top of it. I made several fixes this week and I am planning to do more in coming days.

There is still a decent user community around KornShell and they want to keep it well maintained. I look forward to work with them. Let’s see what we can build together!

This was followed up by a December 2017 post entitled “KornShell – Moving Forward”:

Since my last blog post we have made some decent progress to improve code around ksh93. I will try to summarize recent developments here :

Legacy build system has been deprecated and we are going to move to Meson. The only part of legacy build system that still remains is use of ‘iffe’ (if feature exists) command. It is used for feature detection (same like configure script in Autotools) and will require some extra amount of work for replacement.

Travis integration has been improved. Now we are running test cases on Travis to detect regressions. The new build system allows us to run test cases in parallel. With the legacy build system, building and testing used to take more than 20 minutes, it has been cut down to around 5 minutes now.

We have moved to ‘master’ branch for development. Most of the git projects use ‘master’ for development. But since AST repository contained full source code of all AST projects, it took us some time to reach a conclusion on it. At the end, we decided to move full AST source code under separate branches and use ‘master’ only for ksh93 development.

I want to say thanks to my fellow committer Kurtis Rader for his invaluable contributions. He has been instrumental in improving code and has agitated a few discussions in the community. There is a growing interest in the community to keep ksh93 relevant. We are up to a good start and the list of pull requests and issues is slowly increasing. It is a sign of recovery. KornShell is heading for good times!

So who is “my fellow committer Kurtis Rader”? Well, here is his blog (which by the way is named Dog Is My Copilot. I suggest you read this particular post from his blog. The following quote is from that post:

Why aren’t there any good alternatives to bash or zsh? Specifically, a OS CLI shell that does not suffer from the problems inherent in being compliant with the POSIX.2 (aka POSIX 1003.2) standard? And also doesn’t suffer from the other problem that bash and zsh have due to all the configurable behaviors that make it effectively impossible to predict how those shells will behave?

The impression I get from that post is that he not interested in the long term backward compatibility of a shell. He is unhappy with any shell that complies with POSIX.2.

Here is a quote from another post by Kurtis Rader entitled Is ksh93 still alive?.

The Korn shell lacks many of the features people have come to expect from newer shells. Most notably a good command completion subsystem. But the ksh source code is pretty clean. It has a consistent style and good interfaces. There are things about its style I don’t like such as the use of single statement blocks that are not enclosed in braces since that pattern makes it too easy to introduce a bug and makes it harder to visually parse the code.

Still, at least the code is consistent in employing that pattern. It also omits whitespace around binary operators like minus and commas that separate parameters. At least most of time. Something I think hurts readability especially since it doesn’t do so 100% of the time. I’d probably want to run the code through clang-format and otherwise manually fix the remaining style inconsistencies before contributing more substantive changes. Much like I did for fish. The question is whether the people with commit privileges would accept such changes. And whether they would be open to the idea of implementing some of ideas from newer shells like fish.

Siteshwar Vashisht is the current maintainer of ksh93 at Red Hat. Somehow, he managed to get commit privileges at GitHub for the AT&T/AST repository and he is back porting bug fixes he has made into the ATT/AST repositories while testing these fixes on all the platforms supported by the codebase.

From a post by Siteshwar Vashisht on July 1st 2017 entitled KornShell is not dead:

Despite the lack of pace in development from last couple of years, ksh still remains one of the most popular shells. AT&T released source code on GitHub, but there were no updates from them for more than an year. Last commit was made by AT&T on Jan 11, 2016. I have been submitting bug fixes on the mailing list, but the patches were not added to the upstream repository as there is nobody working on KornShell at AT&T. This week they provided me commit access to the GitHub repository. The latest code was released under beta branch and I am going to build on top of it. I made several fixes this week and I am planning to do more in coming days.

From a post by Siteshwar Vashisht on December 4th 2017 entitled KornShell – Moving forward:

Since my last blog post we have made some decent progress to improve code around ksh93. I will try to summarize recent developments here :

Legacy build system has been deprecated and we are going to move to Meson. The only part of legacy build system that still remains is use of ‘iffe’ (if feature exists) command. It is used for feature detection (same like configure script in Autotools) and will require some extra amount of work for replacement.

Travis integration has been improved. Now we are running test cases on Travis to detect regressions. The new build system allows us to run test cases in parallel. With the legacy build system, building and testing used to take more than 20 minutes, it has been cut down to around 5 minutes now.

We have moved to ‘master’ branch for development. Most of the git projects use ‘master’ for development. But since AST repository contained full source code of all AST projects, it took us some time to reach a conclusion on it. At the end, we decided to move full AST source code under separate branches and use ‘master’ only for ksh93 development.

I want to say thanks to my fellow committer Kurtis Rader for his invaluable contributions. He has been instrumental in improving code and has agitated a few discussions in the community. There is a growing interest in the community to keep ksh93 relevant. We are up to a good start and the list of pull requests and issues is slowly increasing. It is a sign of recovery. KornShell is heading for good times!

I have major concerns about what is happening to ksh93 maintenance and development.

No Steering or Oversight Committee

Siteshwar Vashisht, a relatively junior Red Hat software engineer, teamed up with Kurtis Rader, a software developer who seems to think he knows better than anybody else what changes should be made to the existing build system and codebase. No steering committee was proposed or formed. De facto, Kurtis Rader is in control with Siteshwar Vashisht basking in the sunlight for the moment. He appears to simply commit any changes to the codebase that he wishes to make with no formal code review or peer approval. Frankly, I am amazed that Siteshwar Vashisht’s manager at Red Hat did not require a proper organizational structure to be formed to manage this maintenance and development work.

False Representation

Instead of setting up a new development area on GitHub, or elsewhere, for ksh93.next, the new development is occurring in a repository belonging to AT&T. This gives the totally false impression that AT&T has somehow blessed this new development work and are responsible for such development work – which clearly is not the case. A totally new repository should be created under the ownership of Red Hat or a ksh.next steering committee. Sooner or later AT&T are going to wake up and realize the legal and reputational risk that has occurred.

No commitment to Backward Compatibility

I believe that the overarching principle for ksh.next should be backward comparability wherever possible with the last official release of ksh93. Organizations and companies alike expect a shell to be stable. In many cases 20-40 years of shell script development provide the glue that make their business work flows just silently work.

My distinct impression is that Kurtis Rader, the main committer to date, has zero interest or commitment to code stability or code features that he personally has no experience with, or features that he does not agree with for some philosophical reason.

Totally New Build System

The build configuration system has been changed to Meson (also here) and the actual build system to Ninja. Both require Python 3. Frankly, there was nothing wrong with the existing build system, i.e. iffe and nmake, but Kurtis Rader was not prepared to learn the nuances and internals of the existing build system – so it was disparaged by him and immediately replaced by a build system which he was familiar with.

I predict that this effort will end in tears within a year or so. AT&T will sooner or later act to mitigate their legal and reputational risk. Kurtis Rader will slowly but surely lose the respect and support of the ksh93 community because of his “my way or the highway, I know best, everybody else is wrong” attitude and his ongoing numerous disparaging remarks about the work of the previous maintainers of the codebase. Finally, I suspect that more senior engineers at Red Hat and elsewhere will become aware of the radical changes being made to the ksh93 codebase and push back.

I hope I am wrong… but I doubt it.

Comments are closed.