Very old coder here. Wrote COBOL to help Atari add features to a inventory processing system to account for the fact that "inventory" intially was items received at the loading dock, fork-lifted to the shipping dock and shipped. So "inventory" needed to be booked immediately as sales. Now I dabble mostly with Python and JS/HTML. My memory of the Atari gig was that the most critical part was the CICS code. There was just one guy who knew enough to setup the CICS. If he got hit buy a bus... Well after about a year, the bus would not have mattered. Atari buried millions on unsold carts, and I went from working in a beautiful office complex next to Great America Park, to a windowless basement closet somewhere near Mountain View now making changess because the "forklift inventory" version was no longer needed. I know this is a bit off topic, but "COBOL" was the into I needed.
Ah. The birth era of creative accounting. I remember companies shipped empty CD and booked it as revenue for the quarter. The actual software when ready was delivered as a “patch”.
I was working on CICS in the early 2000s. An insane amount of live COBOL code still moving things around (and algol and Fortran). We occasionally found bugs in code written in the 70s.
I’m not a COBOL dev but I work with mainframes(z/OS).
Most COBOL applications I’ve seen have been banking and insurance related with few exceptions.
Most of them either run as a series of batch jobs or via transaction managers like IMS and CICS.
Backends are usually sequential files(we call them datasets),DB2,VSAM(Virtual Storage Access Method) or DL/1(hierarchical DB that’s part of IMS).
Quite a few places I’ve seen have run IBM MQ as well.
If changes are made to these systems it’s often due to changes in regulation or driven by changes in the business(new financial products being offered etc.
Off-topic: I’ve seen quite a few mainframe related posts on HN fly by over the years. I’ve been meaning to create an account and participate but I’ve only gotten around to it just now.
What is the primary computational task for insurance? Is it computing actuarial tables? Or accounting for the premiums/claims? Or something else? I can excuse banking for not switching off of COBOL but I fail to see how insurance uses it to any significant degree.
Some less-common varieties of life insurance start to look a whole lot like investment products and are (sometimes controversially) sold as such for people in very specific financial situations - think like cash value and death benefit value that vary over time based on asset performance. A legacy insurer selling Variable Universal Life or similar policies would have some financial calculation needs.
And I guess if there’s a whole-of-life policy sold in 1962 that hasn’t been terminated - I guess there’s a lot of grandfathered rules that are just easier to keep in their original systems.
I can’t remark on COBOL, but I did contracting for a time to non-US insurance company replacing their Universe Basic system originally written in the mid 90s, with a .NET Core (3.1 had just come out at the time, around 2019) based system.
The system contained records of all their policies including all the premium factors (e.g. make, model, year of car, parking location, physical address, etc). These premium factors were then fed into a rating engine which would use actuarial tables with custom actuarial defined algorithms to determine premiums.
In insurance companies, working out the correct premium is core to everything. Insurance companies can have lots of different products and their competitive edge comes from how well they structure their offerings and determine the correct premiums based on risk factors. One does not simply rewrite such a thing.
Couple of things I thought were a bit interesting about the place:
- Their single server running Universe Basic (with a hot spare I believe) had 4TB of RAM.
- While I was used to the devs being the stars of the show at the consulting house I worked at, at insurance companies it’s the actuaries.
Accounting is a huge part of it. Sometimes claims are not fully resolved for decades, and the books for that policy hypothetically have to be kept open for as long. This must be kept correct, but is not computationally heavy.
Computing total exposures and possible loss distributions are things which can be more computationally heavy. It includes grouping together similar policies which is multiplicative in complexity.
Insurance is largely about pricing risk. At the heart of most insurance platforms is a risk engine of sorts that takes a list of parameters and produces a premium.
The better your risk models, the more easily you can offer competitive premiums without over-exposure.
(I have worked in insuretech. Although my work was on transactional services and our white labelling capabilities rather than the math heavy stuff)
I think you got most details from the others, but insurance is two basically two different things:
Life Insurance is mostly a savings product, and the insurance part protects you if you live too long.
Property and casualty insurance protects you from losses, including someone's life, but also houses, cars, etc.
The domains are quite different, but they both have specific "insurance business" computing that's related to actuarial science or analysis, i.e. the statistics needed to calculate reserves, policies, prices etc.
I doubt COBOL is used for any actuarial analysis. I think SAS is still strong, but I suspect R is used now. Maybe Python is used in the more static calculations that are handed over to developers, but the actuaries are typically coding whatever they need when they create their models.
The rest is just case management, automating business rules, bookkeeping, payments, and for life insurance also systems for trading securities and funds, and possibly in-house tools for asset management.
There isn't really a strong case for COBOL. The only reason COBOL still is used is that the insurance companies where early adopters and saw computing as a way to reduce the administrative overhead. The investments were made at a time when trusting some hippies running UNIX wasn't really on the table, and even less so trusting some nerds and their rickety PCs. They built up a workforce with COBOL devs that also gained quite a lot of business knowledge.
The digitalisation created another problem - a lot of the older employees were hired to do simple administrative tasks. Even big corporations aren't totally psychopathic so it actually has taken a long time to shift out the employees, and retrain the remainder for the jobs that got more demanding. Even the employees that didn't really have that much high-value domain-specific knowledge to begin with. So the case for more automation was actually not as strong as it could have been.
Even still, although especially life insurance is a totally digital product (damage claims is not), they primarily see IT as a cost centre at heart although they probably claim they do not.
This has shaped their systems and they have tended to replace their old systems when they're forced by external factors, as the upsides - better digital sales, more automated decisions, better trading experience for their customers,etc
are not as easy to achieve as the more tangible administrative automation cost savings they started out with.
Actually, this also applies to banks. You could totally run an insurance company or bank without any mainframes or a single line of COBOL. But the organisations still have COBOL developers and maybe more important an upper management that come from that tradition.
I'm not a mainframe programmer, but coming from x86 land I was very curious. I really learned a lot from the IBM Coursera "Intro to Mainframe" since none of my experience really applied it was tough. It had a real shell account to practice with though.
Not a dumb question at all!
In Europe I’ve seen a few training programs held by companies looking to get new talent in to learn from the older techs. Browse around and see if any companies around you have something like that.
There are some free resources available that will allow you to get training but I haven’t tried them myself. IBM Z Xplore is worth a look as an example: https://www.ibm.com/products/z/resources/zxplore
I hope you find a way in, more mainframe developers and sysadmins(often called systemsprogrammers in the mainframe niche) are always needed.
There's an emulator called Hercules[1] that lets you run (some) mainframe stuff on a PC. There are limits to what you can run on it though, mostly due to licensing issues with IBM.
You can also look at the IBM Redbooks site[2]. Search for terms like Z/OS, MVS, CICS, DB/2, etc. and you'll find a lot of IBM books, whitepapers (well, they call them redpapers, but whatever) and so on.
I've setup Hercules on my Mac before. Fun tool, since I last touched OS2/MVS 40 years ago. Best part is many of the old OS's (MVT, MVS, vm/370) are all still available since IBM never saw the need to copyright them.
> Most COBOL applications I’ve seen have been banking and insurance related with few exceptions.
Banking is a big one yup. I think some countries still use COBOL too for computation related to pension payments.
Best paid consultant I ever met was a COBOL dinosaur. The bank fired him then had to crawl at his feet and accept his insane daily fees when he came back as a consultant.
A very good friend of mine worked at a bank on Java code that would generate... COBOL code. Fun stuff.
My mother and her husband are COBOL devs for a US state government. She works on the health insurance side for teachers and other state employees. Think claim processing.
Lots of batch jobs running at night. Their alert system is an actual human who calls my mom when jobs fail in the middle of the night.
It's high paying for the city they live in, but not high paying for software development. They will both have full retirement and healthcare for life, assuming the government can fulfill it. They are both fully remote since COVID too.
She's also worked for state lottery, teacher's retirement system and DOT.
edit: she says they have a SQL database, but mostly store in IBM IMS
She says it's both. By new stuff, it's mostly one off programs that handle small changes to the way billing or claims are handled. It sounds like they have a library of programs to start with and she extends it to fit the new edge case.
I work with a lot of COBOL dinosaurs in the bank, I often like to watch them work on their 16-colors IBM z/OS host terminals, it's quite mesmerizing. Sometimes they show me some interesting code that was written before I was alive (I'm 36), or tell me stories about big mainframe incidents in the '80s, where they would get called in the middle of the night and flown to a different country to fix a bug because there was no remote desktop back then.
Damn mainframe people flaunting their 16 colors like they're a peacock or something. Shit. We only had one color (and one absence of a color) and that was good enough.
> I often like to watch them work on their 16-colors IBM z/OS host terminals, it's quite mesmerizing
They really are. I had a parttime coworker who moonlighted some mainframe job and he often had another laptop on his desk connected to a z/OS terminal. He would show me some of the jobs and code occasionally too, really fascinating stuff, and he was quite good at it and could navigate quickly.
24-bit color support (\033[38;2;rrr;ggg;bbbm) has been mainline in both Konsole and in libvte-based terminal emulators for many years. 16777216 colors. That's still not every color, but it's every color your monitor can display. When I wrote http://canonical.org/~kragen/sw/dev3/gradient.c in 02016 I had it in Konsole but not libvte, though I think it had been added to libvte upstream. http://canonical.org/~kragen/sw/dev3/gradient.png
Color in terminal emulators was one of the main perks of Linux over other Unixes for me at first!
I’ve built systems for iSeries and none of the modern fancy GUI IDEs come close to the speed of those IBM 5250 terminals. You can still see such terminals in action in POCO baumarkt in Berlin.
One of the modules I saw in action was written before the moon landing, written by a lady programmer.
From what I understand, they log in to a citrix-like webpage called IBM Host On Demand and then start a java applet which is essentially a 16-color 3270 emulator that's connected to z/OS mainframe.
I am working as a mainframe developer at a bank, currently within a kind of data warehouse solution. Mainly writing in COBOL and Java (in USS), with some scripting in Rexx for internal ISPF tools. A lot of SQL as well, we use DB2 as our main database.
Most of our processes are EOD centric, we run a lot of batch jobs (mainly TSO, very little IMS). Integrations are mostly file based but we do both call and expose APIs (“regular REST APIs”) as well as consuming from and producing to Kafka among other things. We integrate with both mainframe and distributed systems on prem as well as “internal” systems hosted on cloud.
We use Git for source control but have a CI/CD solution that is built in house. Quite a lot of other infrastructure is also built in house.
I am mid 30s and am on the younger side looking at the mainframe area as a whole at my employer, however in my team we have several developers is in their 20/30s.
My background is mainly back- and frontend development on the Microsoft tech stack but I have worked, and do work, with whatever is at hand/required. But mostly stuff like .NET and SQL Server on the backend, and Angular/Vue/React on the front end before this.
What is crazy is how simpler is working with mainframes that host nation-wide apps, compared to developing a modern web app..
People joke about old coders brought out of their retirement to maintain a dusty COBOL/RPG program, while the reality is that the tooling is simple enough that a young developer could learn them in a month, and master in less than a year.
Plus, the expertise is not lost after a few years, given the platform focus on incremental improvement, and backwards compatibility.
What are wages like? I heard several years ago that you could land a sweet high paying COBOL gig, but from what I've read recently that doesn't seem to be true?
Met one close to retirement who worked on a ERP system in the food processing industry. Nightly batch jobs would trigger orders from their suppliers, customer service would enter new orders. Two SAP migrations already failed, costing the company millions. All company process knowledge was in code, database fields have been repurposed (but no renamed, too much work), feature development stop long time ago. In parallel a new system was built in-house (no longer trusting external consultants) and his job was explaining what the system does. Probably well paid but he didn't seem to care, he just wanted to work less and retire on good terms.
I quickly worked at a company like that, that had large parts of their core business logic running on an AS400, and they were asking what they need to do to migrate to something newer - was surreal. A few hundred employees all interacting with the system - in building A, a deliver arrives, they scan it like this, press this button. Building B, a guys job was to oversee the conversion of tables from As400 to a csv format suitable for some other outsourced software. Data goes into system C and shows to employees working on conveyor belts this and that. Hundreds of kilometers away, truck drivers get a notification for this and that.b
And, nobody knew how the whole worked. Everyone has their niche of interaction with the system. They would be able to shave off an insane percentage off expenses (in the form of employees whose job exists for no real reason), but the switching costs would also be immense.
I sometimes wonder what came of their company. The system was so far beyond the complexity that anyone could grasp, they had no inhouse devs, they'd need people with the competency to judge which competency they need.
For sure; I'll believe that an AI can read and "understand" code, extract meaning and requirements from it, but it won't be the same as a human that knows requirements.
Then again, a human won't know all requirements either; over time, requirements are literally encoded.
Not a COBOL developer, but working at a sizeable bank I witnessed the phasing out of their mainframes and AS400 systems. They ran some critical systems, both in retail and wholesale banking. They either converted to java, and optimized that code, but some COBOL code from the mainframe, and all of the AS400 stuff was converted into Micro Focus COBOL, which runs on Windows, which could be hosted on our Private Cloud. I worked on helping them migrate to our cloud infra, which was an interesting exercise. There was a very tangible cultural gap between the people maintaining and developing these applications and the rest of the organization.
In my experience, it's usually lack of awareness about modern security risks, and lack of familiarity with modern infrastructure paradigms. The latter really isn't a problem since these systems are usually standalone, but the former does become a problem - they often are from a time where this just wasn't something to consider. As a result, these legacy systems are often using default passwords, have tons of crazy stuff exposed to the network, and are comprised of custom code written specifically for the business purpose (so the documentation is only as good as what they made).
On the other hand, these guys generally write pretty neat, lean code that is quick, reliable, and directly responsive to the business. The really fun thing is watching the users fly through the keyboard-only screens, sometimes with muscle memory that is faster than the terminal emulator can update - they're literally working ahead of the screens.
Oh yes, I remember that when we swapped out a bunch of terminals at an airline.. The users complained it was all way too slow on the new Windows machines with MS SNA server in between... I was wondering what it was all about, as a young and very naive dropout from uni on his first IT job. When I came down, this dude was banging on his keyboard and after some time stopped, pointed at the screen and you could see it slowly catching up, screen by screen.. He showed me the directly connected version next. I learned something that day.
That's awesome. I set up Arch Linux a while ago, and despite working in Linux shops for more than a decade, let's just say I was very out of my element...
Reminds of me of a TUI Banking software that ran on Sun Solaris. It could keep up as fast as you can navigate - few months in and you could fly through the screens. Then it was "upgraded" to a web-based version and all of us were up in arms, it was like being downgraded to a tractor after experiencing a racecar.
As a support engineer at IBM we used a mainframe system called NRCPMA iirc... I think NR stood for Northern Region. Accessed via a terminal emulator, fully customizable with macros, fastest tool I ever worked with indeed, once you climbed the initial learning curve.
Reminds me of modern IDEs -- developers, both old and new, are too lazy to learn a complex IDE to speed up their work, even though it's their main tool for making money.
I don't think efficiency of navigating your IDE is a major factor in your productivity. If you like your setup, it's probably not going to matter a whole lot. The tool you make money with is your brain.
1. Navigating the code, e.g easily see all the callers, navigate up/down the call tree requires static code analysis. Super handy while reading someone's else code, which is like 90% on large projects.
2. Quick refactorings. Often times I see people discuss in lengths what would/could be instead of just go and try it out quickly, seeing all the pros and cons. Many times I proven myself wrong by trying it out and seeing pitfalls I didn't see earlier.
3. Warnings: so many real bugs could've been prevented if developers had seen (or cared about) to IDE showing a warning. Many PR review suggestions are detectable by a proper IDE without wasting reviewer's time.
4. Hotkeys (what the parent comment was talking about) -- speeds up all of that, especially refactorings, freeing dev's brain for thinking of architecture and other problems.
I can go on an on. Sometimes it feels like 50%+ of AI usage for coding is to free up fingers, not knowing that they were already mostly free by using static analysis features/hotkeys.
In my experience mainframes at financial institutions are hidden behind IBM middleboxes that are specifically designed to obviate the infrastructure risks. It's a classic example of a company selling you both the problem and solution.
That's just an example of incremental improvement. Mainframes and midranges adapt with the times without losing what works. Modern midranges, for example, can run C, Python, bash, and web servers.
I would say these people were in a relationship with the mainframe, if that makes sense. And also having worked at IBM in the past where I sat adjacent to the mainframe support team for Business Services, I totally get it. Mainframes are awesome if you ask me, and in a sense we have been trying to reinvent a lot of its goodness with "commodity" x86 hardware.
From a technical-cultural perspective it was mostly sulkiness, and a complete and utter lack of embracing the paradigms of distributed computing. Also, like most internal clouds, there were plenty of issues as it was. Initially they just tried to replace mainframe application components 1:1 onto VMs in whatever way and whenever anything was <100% reliable they complained that our cloud was not able to do it. I had to explain in a very harsh way, under a lot of pressure (I believe not hitting the deadline of switching off the mainframes meant renewal for a year at 40 Mil.. or thereabouts) the realities of "cloud".
The developers I spoke with in that time though, were very much the opposite of the move fast breaking things crowd. Intelligent, but also narrow minded I would say.
Glad to meet y'all. Yes. I code COBOL along with C, C++, Java, REXX, SQL, and IBM mainframe Assembler. I support the racehorse of IBM mainframe operating systems, z/TPF. There are few familiar with it. Think of z/OS as a Clydesdale horse, big and SLOW.
zTPF runs the airlines, the model for real online transaction systems... Not SQL relational database shopping carts.
Botton line, everything is Assembler. It's all just bits.
Heh I worked one of these. We handled arithmetic in the DB tho. Lot of PL/pgSQL running under the typescript, TS was more of like a middleware or API layer for things that could change more frequently. Finance code is to some extent transcribing of regulation & law into code and we kept all that in postgres.
Indeed, I've worked on billing system that relied heavily on pure JavaScript. Not even modern flavors with map/reduce, etc. - ECMAScript 5. It worked surprisingly well and our bottleneck wasn't the runtime but rather the databases we were constantly upserting to.
It sounds kinda crazy but with good change control, documentation, good relationship with the ETL team - it was pretty maintainable.
I met a dev who's mom had been working on legacy banking systems her whole career. She had started in the eighties and she still did some urgent jobs at a crazy rate despite officially having retired.
My stepmom who retired five years ago, did COBOL dev as part of her banking job until 2002ish and then she was full-time management track. In her bank, most of the work had been integrated with Java, and the Java was done by outsourced Indian teams. At the time she retired she felt the Indian teams had been failing for years to meet objectives, and finally management was seeing it. Additionally everybody who knew the COBOL side of things was retiring at the same time as she was and she did not want to know what the system would look like in five years.
My mother used to teach Cobol back in the 80’s in Brazil but later she transitioned into management and haven’t touched a line of code for more than 30 years, she can’t even speak english wtf
I haven't worked in COBOL, but I've worked with it.
This was around 1999, and I was building a system for configuring and ordering custom PC's at a large distribution company. One of the feature requirements was that we display inventory over the various options. (ie: There are 373 20G disks in stock, but only 12 30G disks in stock). The idea was that this would let a customer ordering 200 machines know that they should pick the 20G disk if the wanted it now.
The way inventory at this company was done was via two systems. There was a normal SQL database that had access to a daily snapshot of inventory taken from a mainframe that always had the up to date data. With the mainframe taking a while to process queries, we used the less current SQL database for the majority of the UI, but took the time to query the mainframe once a customer was in the shopping cart. Customers might see a change during the flow, but it would at least let them see the most current data prior to committing to a purchase.
The mainframe query itself was implemented by someone else as a COBOL job that produced the required inventory numbers. From my point of view, it was just a limited sort of query conducted over a specialized JDBC driver. (Not necessarily the weirdest aspect of that design.... for a variety of reasons, we did the UI in ASP/VBScript, the back end using Microsoft's JVM invoked via COM Automation, and the SQL database link was via a dubious use of a JDBC/ODBC bridge to connect to SQL Server. It all worked, but not the most solid architecture.)
==
My only other mainframe experience was working as an intern for a utility company a few years prior (1991-1992). They used CDC Cyber mainframes to run the power grid, using something like 4MM lines of FORTRAN code. The dispatchers themselves interfaced to the system using consoles with 4 19" color displays running at 1280x1024. Heady stuff for 1991. (The real time weather radar screen was cool too, in an age before the internet made it common place.)
Not Recently. 2010ish was working for an insurance processor that was actively writing thousands of lines and executing on z/OS with no plans to migrate.
I was part of team that was writing web applications that needed to call z/OS transactions. The IBM solution was to use their Transaction Gateway product, which cost a ton, and was slow as shit. We developed a framework that used annotations on the Java Side to map COBOL Records to Java Objects and invoke transactions over a TCP socket. Learning how to pack decimals and convert encodings was pretty cool. We ended up with a framework that was at least a zillion times faster than the IBM solution. Left that job though as the company was is distress and was losing customers (health plans). They eventually folded.
All I do is COBOL. Everyday. Our team has led over 40 “modernization” efforts, with about half migrating the apps to an X86 platform. There are “tons” of COBOL apps in Production, and not just limited to Financial organizations. I work with County, State and Federal systems. Retail, Insurance, Telecom, and Manufacturing.
It’s not uncommon to work on programs that were original written in the 80’s or 90’s
The IBM Z is feature rich, but expensive and limited by software choices for apps not running ZLinux. (Think virtual Linux server)
The biggest catalyst now is the increasing license cost of mainframe third party software
I have written cobol in the past. I worked at a financial company that had a decent amount of cobol devs around, with the primary database being DB2. The code is mostly financial transactions and record updates, basically CRUD code. Cobol was essentially the backend with the front end being Java and Javascript/Angular.
My father is 75 and he still works, has his own software development company with his own back-office program written in cobol. He started a company back in 1991 with two other cobol programmers, they are retired now, and while almost all of the code they wrote has been replaced with c# code by younger programmers there are still some parts of the code written in cobol that he still maintains.
Random OT question: I was raised by, erm, relatively uneducated folks. Is there anything especially great about having a software programmer for a father? (As my kids have one)
They may eventually learn the valuable skill of politely smiling and nodding as someone talks with great passion about things with zero relevance to their day-to-day life, assuming they don't shut up about it at home.
Actually, tbf, my daughter's pretty interested in my coding and electronics projects. She picks things up alarmingly fast. I taught her a practically-one-way encoding scheme for passwords I've had incredible trouble teaching anyone else (LLMs also can't figure it out), and she completely understood it after the second example I gave and even added her own extra twist to it. Her passwords now are both memorable and extremely secure against dictionary attacks even with any mutation scheme someone could reasonably imagine.
-So I think that should probably go in the plus column.
Having your kids also be technical and bail you out of the shit you've built for yourself. Or at least nudge you away from bad ideas, do listen to them.
I'm being a little unkind to my Dad. He moved to management fairly early on and didn't really keep up with things.
He taught me a hell of a lot though, and did really know his shit at one point. It worries me how much his skills and understanding have declined over the years.
My brother works with COBOL for a bank here in Brazil, he is young (in his 20s), started before finishing his degree. Pay is poor, hours are insane, he is overworked as hell, and anything “modern”, like git, is out of the question.
He’s trying to learn Go now and modernize himself to see if he can get out. I’m trying to help as much as I can. Hopefully, he’ll land a job somewhere else this year.
at the banking place I work running things in k8s, z/OS-people are actually the ones running custom git clients in go on z/OS. Bonus: they have no nosy Java devs (recurringly producing threading bugs...) saying "but we all use spring boot!!!" and likely no manager asking "is this cloud-ready???"
What I'm wondering: Are the salaries high? Not just because you've been employed at the job for a long time with regular raises, but because it's hard to find developers.
No the salaries aren't high. They are typically lower than other software engineer salaries. There are a large number of contractors from Indian consulting companies with "experience in cobol" to make run of the mill cobol cheap enough.
The very high salaries you hear about sometimes are always for VERY specific mainframes that are extremely old with a lot of quirks, and are usually being paid to consultants.
> There are a large number of contractors from Indian consulting companies with "experience in cobol" to make run of the mill cobol cheap enough.
Seeing the horrible performance from Indian offshore firms with modern languages I cannot imagine the mess they make with legacy languages like Cobol. Or is it the other way around?
But the code still has to work. LLC's and other corporate structures only protect the owners if the company goes bankrupt, which it will if its systems stop working.
Ditto with market control, it's not some permanent crown you achieve. Companies have to keep performing to keep their market share.
E.g., if you opened an account at a major bank, and your transactions started failing, would you keep banking there?
> E.g., if you opened an account at a major bank, and your transactions started failing, would you keep banking there?
A lot of people who land in that situation do continue banking there since they are either tied into that bank through loans/debt, or lack the time/energy to move elsewhere.
A while back I came across job listings for a COBOL consultancy near me that only seems to hire fresh grads for well below market rate (not much higher than retail/restaurant jobs - this is in a cheaper part of the US). They promised to train their employees from the ground up and implied that COBOL knowledge would set them up for a really profitable career. It seems like they were taking advantage of the common advice: "just become a COBOL developer, it pays well because nobody wants to use COBOL!" But I'm skeptical that someone coming out of that consultancy with 2 or 3 years of experience in nothing but COBOL would do well on the job market.
The places I know that use (or used cobol 5 years ago) were all in hiring freezes for cobol developers and were trying to get off of it as much as they could (no new development, only maintenance, etc). I don't think its a surefire bet.
I have a friend who works writing COBOL for a US state government, I sent him this post and this was his response:
I work in COBOL for government systems.
We upgraded our old mainframes to IBM Z16's, and we just got some new ones in recently for our backup server.
Part of the job is taking whatever they decide in legislation and translating that into code that processes whatever they decided to make law, from fees, suspensions, special legislation for certain areas, etc.
Our programming environment primarily uses TSO/ISPF and changeman for version control. We have access to IBM's IDz (previously RDz) as an IDE for development, and I would personally prefer to use that but haven't been able to get it installed on my work computer due to licensing issues. Part of security protocol is that we cannot use anything open-source, so VSCode with Zowe is, unfortunately, out of the picture.
We maintain a lot of old programs and modules, but we are also actively developing new programs as we expand our IT department - and yes, that is new COBOL programs. We have a Linux side as well which mainly deals with the web-side, but they still interact with the mainframe to call on modules - but all they are really doing is sending data to CICS to get data back. They do not know anything about the COBOL itself or how to program in it.
In undergrad(around 2005) we used to have a COBOL lab, most of the assignments were pretty much add/update/delete types. A friend of mine found COBOL so boring, in order to keep it interesting he wrote COBOL program for brothel(not real one), to maintain the customers, prostitutes, transactions and whole nine yard. Despite of not liking the language, he was the best COBOL programmer in college.
A legitimate question, but so far not many answers, and they're mostly from people who know people who know COBOL devs. This is to be expected.
Demographically, COBOL devs skew older, and there aren't a lot of graybeards left on HN. This place used to be full of them, and they always had interesting and unusual insights and techniques to share. Those days are long gone.
IMO, Graybeards have largely left HN for a few reasons:
- They're tired of being shouted down by the Reddit-quality ageism that lingers through this forum.
- They're mature enough to no longer be interested in chasing every little tech fad as if their lives depended on it, and that's 90% of what HN has become.
- As most older people do, have other things in their lives that are more interesting than work. Family. Children. Hobbies. Traveling. Service. The world is full of things more rewarding than being terminally online, or being reminded of your day job.
I applaud your curiosity, but you're standing in a church asking, "Where are all the atheists?" COBOL devs aren't here. And where they are is likely not online.
The tinkers are on vcfed. All of the others I know (not many, maybe a dozen) are barely online outside of work. They're largely bored with the internet.
This is in my past, so no COBOL recently but in one of my first consulting gigs I wrote several business programs in RPG III. Which is the nightmare that you might imagine it is. Think plugboards. Halfway through the IBM guy came by and installed COBOL and wow what a difference. My last act was to recommend that they get a computer department. And so Tom Monaghan called IBM
Apropos of nothing in particular, there's an older HN thread about "Good resources for learning COBOL" that some folks here might find interesting. OK, calling it a "thread" is over-stating things, but still...
Actively working on multiple COBOL systems in the Healthcare field, under the umbrella of “Patient Accounting”. Still developing new programs and making modifications. I’d say these applications will be active for at least another 5 years!
For my sins, I work in Education supporting PeopleSoft. That means that I do work in COBOL on occasion. I am not tasked with writing anything new in COBOL, but I so quite a bit of analysis and support. That is: I _read_ COBOL more than I write it.
There are three flavors of COBOL that I deal with: PeopleSoft delivered, Vendor delivered, and University modified. Most of the work I do in COBOL breaks down to reading the code to determine why a given behavior is observed. Only once (in University modified code) have I needed to make an actual edit. The rest of the times I either modify the flow of information into the COBOL code or I modify the resultant after the code has run.
Worked at a large insurance company in the late 90s where leadership touted a big benefit of converting to PeopleSoft Financials as part of their Y2K remediation was the elimination of their dependency on the archaic COBOL language, blissfully unaware that their new PeopleSoft applications were using Micro Focus COBOL under the covers. Oops! https://blogs.oracle.com/peoplesoft/post/take-note-significa...
Going on thirty years ago, I had some experience with Peoplesoft. I'm interested to hear that it still uses COBOL, though on the other hand, why shouldn't it?
I am not a COBOL dev but have worked on a few codes here and there. I played around with LLM and it appears to be not too bad at COBOL, what’s the consensus or your experience utilizing LLM for COBOL?
I worked as a cobol dev in my first fulltime job. I did mostly cobol85 (microfocus) and object-cobol (netexpress/fujitsu cobol), with isam files. Did the whole "migrate to year 2000" process in a couple of big but old applications (healthcare management software and manufacturing management software). Cobol is a school by itself; there are so many ways you can shoot yourself in the foot that one either learn how to structure programs avoid a spaguetti mess, or slowly descend into madness trying to fix bugs :)
Government. About 25% of my job. No day to day mainframe development, but we do need to update some logic for new policies and regulations.
There’s not much maintenance work. There are very few bugs, as the core applications have been running for decades, most come up with interactions to external services.
Any major development projects are only in service of lower overall COBOL development time, like transitioning some business logic to database updates.
And there is a decommission plan for the mainframe, so plenty of work helping that team.
I worked in an IBM mainframe shop as my first job. The COBOL devs for the most part did not participate in programming/tech communities. It was a 9-5 job and they didn't think about it otherwise. Maybe they would have a subscription to Datamation magazine or something like that but I doubt it.
But keep in mind there was no internet, no podcasts, no youtube. If you needed to learn something you learned it at work, on work time. If something new was introduced, IBM came in and conducted training, at work. For something big you might get sent to an IBM training center for a week. There was no need for (and no real way to do) any learning on your own outside of work.
I don't code in Cobol myself anymore but I manage team who does.
We work on Phoenix, government of Canada payroll system. If you google it up, you'll see some interesting coverage. However, the underlying peoplesoft ERP itself is rock solid at every other client I've served over last 25 years. Peoplesoft uses Cobol and sqr, as well as proprietary languages stored in database, application engine and peoplecode.
Key payroll processes are in Cobol. This is because of its tight integration with database and ability to manually control database cursors. We are very database oriented when it comes to performance. Our developers need to know the programming language, but also deep understanding of client business processes, and sql optimization. They also work closely with our dbas to ensure good performance. So our developers are technically proficient in Cobol and couple of other languages, but also very very strong in sql optimization, and understand clients payroll rules and can speak intelligently with compensation advisers and payroll processors.
I personally found that to be true for most Cobol programmers - whereas typical hacker news Dev seems very technology oriented and frequently moving, typical Cobol programmer is very business process aware and integrated with corporate line of business. They don't move as much for several reasons, but that deep awareness of client is one of them.
Edit: I shoild mention, while peoplesoft can and does work on mainframe, most of my clients are on windows, Linux, or AIX. COBOL is not quite as mainframe specific as it sometimes seems :-). See e.g.microfocus Cobol for a modern multi platform compiler.
Not COBOL but years ago I did some work on IBM AS/400 using RPG. For those who don’t know, RPG was originally written (in 1959) for punch cards and the programs had to be written in a pattern that resembles punchcard. It was a fun experience and I am grateful to have dabbled in it.
Many megacorps still run AS/400 and it’s uptimes and performance is legendary.
Edit: Forgot to mention that I was mentored by folks more than twice my age that time.
Not a COBOL dev myself but for many years I worked in government healthcare for my state. Probably 75% of developers and BAs for the organization were working to support an extremely large mainframe system(s). I worked as a web developer and we had to use a product called Hostbridge that allowed our web applications to interface with the mainframe via javascript.
I've done COBOL since the 80s, but I don't have a paying gig for it anymore. That said, I have situations where I need to process a flat file of data, and I find it a lot easier to just fire up gnucobol and write the program than try to remember the syntax for python or golang.
Followup question: what’s it like to write COBOL? It’s so ancient as this point you never hear anyone actually talking about what the developer experience is like. How performant is it? What features does it have (or not have)? How bad is it really?
I was a consultant at a very large insurance company in the 90s when the dictate came from the top “no more assembler”. There were a few groans from the audience of developers.
Not me, but not even 5 years ago one of my friends was working on Cobol codebase running on IBM zOS. It did not last long, but as far as know they had a decent time.
Ex COBOL dev here. Started at NCR on COBOL for minis and mainframes. Then built stuff on Wang VS COBOL for ERP systems until 93. Quite a few other languages since then but been a Product Manager for past 20+ years.
Same here, I worked on Wang VS COBOL too. IMO, it had the best TUI Screen generation Tools I have ever seen.
Towards the end, I worked on a project to port Wang VS OS and its COBOL to AIX. I was tasked finding issues with COBOL Programs we had on the VS. It was a good environment, but Wang went CPT 11 before it was ready :) It was rather close to being complete.
This is oddly true in my experience. I ask people where they get their information from and the channels are generally a lot more sparse than topics posted to HN.
If I were to venture to guess: I'd say "most developers" seek work-life balance and aren't interested in reading long-form articles on how to do X or Y when they are off the clock.
Y'all commented about the number of terminal colors. That's funny. Only 1 color is needed for programming. The IBM mainframe terminals (3278) were called Green screens for a reason.
Work on a dual Java/COBOL team in a F500. Not a COBOL dev myself but I'm surrounded by them. Two biggest problems are:
a) the systems are very tightly coupled, like uber-monolith architecture, and it's hard to QA them without end-to-end testing through everything. Good luck getting anyone to refactor them because a1) they're going away and a2) they're so huge. Which leads into . . .
b) there's 40 years of undocumented business logic in there, or business logic that only exists in code comments. And it still makes billions of dollars a year. So good luck replicating 40 years of tribal knowledge in the new systems.
c) It was written by cowboy coders 40 years ago when the company was a startup, so no one can learn to work on it without first getting hired here. The joke is one of the original architects went and created his own dialect of COBOL.
I imagine it's the one place where LLMs would absolutely shine. COBOL jobs are usually very verbose, lots of boilerplate, but what they do is mostly very straightforward batch processing. It's ripe for automation with LLMs.
The flip side is that banks are usually very conservative about technology (for good reason).
I don't think LLMs are anything human language specific, so would they really shine here? I.e, COBOL and SQL may be great for humans who otherwise aren't used to programming languages, but LLMs have seen everything, and thus are able to know any (programming) language, not just ones which are English-like.
I think it would shine because COBOL is very verbose and the programs written with it (batch jobs) very straightforward but also very boilerplate-rish and boring to write by hand, a situation with little risk and high reward to automate.
IMO the ideal path to working on COBOL without having decades of experience would be to spend a few days groking the syntax and writing toy programs for practice, then collaborate with a large LLM to understand the current code and gradually make changes.
Have you worked with COBOL? My understanding is that the language itself isn't the real problem. Mainframes, control languages: everything around COBOL is very different from most Windows/UNIX people have experienced.
Yep. I have recently used prompts to ask for COBOL code solution so I can compare it to other languages I also know to check quality of the answer. So far no mistakes.
LLMs are terrible at brainfuck. I spent a solid week attempting to use generative models to iteratively refine BF program tapes with nothing to show for it. I've written genetic programming routines that can produce better brainfuck programs than ChatGPT can.
For example, if I prompt ChatGPT: "Write me a BF program that produces the alphabet, but inverts the position of J & K" it will deterministically fail. I've never even seen one that produces the alphabet the normal way. I can run a GP algorithm over an example of the altered alphabet string and use simple MSE to get it to evolve a BF program that actually emits the expected output.
The BPE tokenizer seems like a big part of the problem when considering the byte-per-instruction model, but fundamentally I don't think there is a happy path even if we didn't need to tokenize the corpus. The expressiveness of the language is virtually non-existent. Namespaces, type names, member names, attributes, etc., are a huge part of what allows for a LLM to lock on to the desired outcome. Getting even one byte wrong is catastrophic for the program's meaning. You can get a lot of bytes wrong in C/C++/C#/Java/Go/etc. (e.g. member names) and still have the function do exactly the same thing.
Very old coder here. Wrote COBOL to help Atari add features to a inventory processing system to account for the fact that "inventory" intially was items received at the loading dock, fork-lifted to the shipping dock and shipped. So "inventory" needed to be booked immediately as sales. Now I dabble mostly with Python and JS/HTML. My memory of the Atari gig was that the most critical part was the CICS code. There was just one guy who knew enough to setup the CICS. If he got hit buy a bus... Well after about a year, the bus would not have mattered. Atari buried millions on unsold carts, and I went from working in a beautiful office complex next to Great America Park, to a windowless basement closet somewhere near Mountain View now making changess because the "forklift inventory" version was no longer needed. I know this is a bit off topic, but "COBOL" was the into I needed.
Ah. The birth era of creative accounting. I remember companies shipped empty CD and booked it as revenue for the quarter. The actual software when ready was delivered as a “patch”.
Surely creative accounting is a time honored tradition?
Seems there is a GAAP in their books
I was working on CICS in the early 2000s. An insane amount of live COBOL code still moving things around (and algol and Fortran). We occasionally found bugs in code written in the 70s.
I’m not a COBOL dev but I work with mainframes(z/OS). Most COBOL applications I’ve seen have been banking and insurance related with few exceptions. Most of them either run as a series of batch jobs or via transaction managers like IMS and CICS. Backends are usually sequential files(we call them datasets),DB2,VSAM(Virtual Storage Access Method) or DL/1(hierarchical DB that’s part of IMS). Quite a few places I’ve seen have run IBM MQ as well.
If changes are made to these systems it’s often due to changes in regulation or driven by changes in the business(new financial products being offered etc.
Off-topic: I’ve seen quite a few mainframe related posts on HN fly by over the years. I’ve been meaning to create an account and participate but I’ve only gotten around to it just now.
What is the primary computational task for insurance? Is it computing actuarial tables? Or accounting for the premiums/claims? Or something else? I can excuse banking for not switching off of COBOL but I fail to see how insurance uses it to any significant degree.
Some less-common varieties of life insurance start to look a whole lot like investment products and are (sometimes controversially) sold as such for people in very specific financial situations - think like cash value and death benefit value that vary over time based on asset performance. A legacy insurer selling Variable Universal Life or similar policies would have some financial calculation needs.
And I guess if there’s a whole-of-life policy sold in 1962 that hasn’t been terminated - I guess there’s a lot of grandfathered rules that are just easier to keep in their original systems.
I can’t remark on COBOL, but I did contracting for a time to non-US insurance company replacing their Universe Basic system originally written in the mid 90s, with a .NET Core (3.1 had just come out at the time, around 2019) based system.
The system contained records of all their policies including all the premium factors (e.g. make, model, year of car, parking location, physical address, etc). These premium factors were then fed into a rating engine which would use actuarial tables with custom actuarial defined algorithms to determine premiums.
In insurance companies, working out the correct premium is core to everything. Insurance companies can have lots of different products and their competitive edge comes from how well they structure their offerings and determine the correct premiums based on risk factors. One does not simply rewrite such a thing.
Couple of things I thought were a bit interesting about the place:
- Their single server running Universe Basic (with a hot spare I believe) had 4TB of RAM. - While I was used to the devs being the stars of the show at the consulting house I worked at, at insurance companies it’s the actuaries.
Accounting is a huge part of it. Sometimes claims are not fully resolved for decades, and the books for that policy hypothetically have to be kept open for as long. This must be kept correct, but is not computationally heavy.
Computing total exposures and possible loss distributions are things which can be more computationally heavy. It includes grouping together similar policies which is multiplicative in complexity.
Insurance is largely about pricing risk. At the heart of most insurance platforms is a risk engine of sorts that takes a list of parameters and produces a premium.
The better your risk models, the more easily you can offer competitive premiums without over-exposure.
(I have worked in insuretech. Although my work was on transactional services and our white labelling capabilities rather than the math heavy stuff)
I think you got most details from the others, but insurance is two basically two different things:
Life Insurance is mostly a savings product, and the insurance part protects you if you live too long.
Property and casualty insurance protects you from losses, including someone's life, but also houses, cars, etc.
The domains are quite different, but they both have specific "insurance business" computing that's related to actuarial science or analysis, i.e. the statistics needed to calculate reserves, policies, prices etc.
I doubt COBOL is used for any actuarial analysis. I think SAS is still strong, but I suspect R is used now. Maybe Python is used in the more static calculations that are handed over to developers, but the actuaries are typically coding whatever they need when they create their models.
The rest is just case management, automating business rules, bookkeeping, payments, and for life insurance also systems for trading securities and funds, and possibly in-house tools for asset management.
There isn't really a strong case for COBOL. The only reason COBOL still is used is that the insurance companies where early adopters and saw computing as a way to reduce the administrative overhead. The investments were made at a time when trusting some hippies running UNIX wasn't really on the table, and even less so trusting some nerds and their rickety PCs. They built up a workforce with COBOL devs that also gained quite a lot of business knowledge.
The digitalisation created another problem - a lot of the older employees were hired to do simple administrative tasks. Even big corporations aren't totally psychopathic so it actually has taken a long time to shift out the employees, and retrain the remainder for the jobs that got more demanding. Even the employees that didn't really have that much high-value domain-specific knowledge to begin with. So the case for more automation was actually not as strong as it could have been.
Even still, although especially life insurance is a totally digital product (damage claims is not), they primarily see IT as a cost centre at heart although they probably claim they do not.
This has shaped their systems and they have tended to replace their old systems when they're forced by external factors, as the upsides - better digital sales, more automated decisions, better trading experience for their customers,etc are not as easy to achieve as the more tangible administrative automation cost savings they started out with.
Actually, this also applies to banks. You could totally run an insurance company or bank without any mainframes or a single line of COBOL. But the organisations still have COBOL developers and maybe more important an upper management that come from that tradition.
Thanks for your insight - it’s comments like this which make HN a place worth visiting every day.
And welcome!
Dumb question: mainframes and z/OS look interesting, how does one get started with learning about those systems and those environments ?
I'm not a mainframe programmer, but coming from x86 land I was very curious. I really learned a lot from the IBM Coursera "Intro to Mainframe" since none of my experience really applied it was tough. It had a real shell account to practice with though.
https://www.coursera.org/professional-certificates/ibm-z-mai...
Also the MOSHIX mainframe YouTube channel has a lot of info, and helped me setup HERCULES emulator for my own little mainframe experience.
http://www.hercules-390.org/
Not a dumb question at all! In Europe I’ve seen a few training programs held by companies looking to get new talent in to learn from the older techs. Browse around and see if any companies around you have something like that.
There are some free resources available that will allow you to get training but I haven’t tried them myself. IBM Z Xplore is worth a look as an example: https://www.ibm.com/products/z/resources/zxplore
I hope you find a way in, more mainframe developers and sysadmins(often called systemsprogrammers in the mainframe niche) are always needed.
Edit*: Spelling and grammar
I’ve run through a bunch of ZXplore and it’s more fun than I expected.
They have it hooked in to VS Code now. It’s weirdly modern. And you get to play on a real z machine.
Recommendable summer/holiday tinkering project. It’s amazing how much and yet how little has changed in computing and transaction processing.
There's an emulator called Hercules[1] that lets you run (some) mainframe stuff on a PC. There are limits to what you can run on it though, mostly due to licensing issues with IBM.
You can also look at the IBM Redbooks site[2]. Search for terms like Z/OS, MVS, CICS, DB/2, etc. and you'll find a lot of IBM books, whitepapers (well, they call them redpapers, but whatever) and so on.
[1]: https://en.wikipedia.org/wiki/Hercules_(emulator)
[2]: https://www.redbooks.ibm.com/
I've setup Hercules on my Mac before. Fun tool, since I last touched OS2/MVS 40 years ago. Best part is many of the old OS's (MVT, MVS, vm/370) are all still available since IBM never saw the need to copyright them.
This is a good article on the subject: https://arstechnica.com/information-technology/2023/07/the-i...
Do you think there’s any money in helping businesses modernise their stack as a consultant?
It would be kinda cool if you’d create a CICS Hacker News UI. ;)
nice, welcome to the party
Thank you!
> Most COBOL applications I’ve seen have been banking and insurance related with few exceptions.
Banking is a big one yup. I think some countries still use COBOL too for computation related to pension payments.
Best paid consultant I ever met was a COBOL dinosaur. The bank fired him then had to crawl at his feet and accept his insane daily fees when he came back as a consultant.
A very good friend of mine worked at a bank on Java code that would generate... COBOL code. Fun stuff.
My mother and her husband are COBOL devs for a US state government. She works on the health insurance side for teachers and other state employees. Think claim processing.
Lots of batch jobs running at night. Their alert system is an actual human who calls my mom when jobs fail in the middle of the night.
It's high paying for the city they live in, but not high paying for software development. They will both have full retirement and healthcare for life, assuming the government can fulfill it. They are both fully remote since COVID too.
She's also worked for state lottery, teacher's retirement system and DOT.
edit: she says they have a SQL database, but mostly store in IBM IMS
Is it entirely maintenance or does she also build new stuff with COBOL?
She says it's both. By new stuff, it's mostly one off programs that handle small changes to the way billing or claims are handled. It sounds like they have a library of programs to start with and she extends it to fit the new edge case.
By state, you mean US state, right?
Correct. Updated.
Very likely.
I work with a lot of COBOL dinosaurs in the bank, I often like to watch them work on their 16-colors IBM z/OS host terminals, it's quite mesmerizing. Sometimes they show me some interesting code that was written before I was alive (I'm 36), or tell me stories about big mainframe incidents in the '80s, where they would get called in the middle of the night and flown to a different country to fix a bug because there was no remote desktop back then.
Damn mainframe people flaunting their 16 colors like they're a peacock or something. Shit. We only had one color (and one absence of a color) and that was good enough.
16 colors is juvenile. When I was a child, I was taught arithmetic using colored rods. I grew up and today I use monochromatic screens.
(paraphrased: https://groups.google.com/g/golang-nuts/c/hJHCAaiL0so/m/kG3B...)
OT: Do you know if was he being serious, joking, or "haha only serious" in the linked message?
i think that's at least mostly serious. he's not a fan of syntax highlighting.
https://ctrl-c.us/posts/highlight
Most of the comments by everyone in that thread are sarcastic, and the tone of the thread is lighthearted.
Pike genuinely doesn't seem to care for syntax highlighting, though understands that his is a minority opinion.
I work with developers who code on OS 2200 and VAX VMS (running on Charon emulation).
Fun fact: the first SMP UNIX implementation ran on top of EXEC 8, the kernel of OS 2200.
"Any configuration supplied by Sperry, including multiprocessor ones, can run the UNIX system."
https://www.nokia.com/bell-labs/about/dennis-m-ritchie/retro...
Edit: https://web.archive.org/web/20150611114648/https://www.bell-...
https://en.m.wikipedia.org/wiki/UNIVAC_1100/2200_series
> I often like to watch them work on their 16-colors IBM z/OS host terminals, it's quite mesmerizing
They really are. I had a parttime coworker who moonlighted some mainframe job and he often had another laptop on his desk connected to a z/OS terminal. He would show me some of the jobs and code occasionally too, really fascinating stuff, and he was quite good at it and could navigate quickly.
I used to work with the most amazing blind COBOL programmer on CICS code. His speed in finding and fixing code with a screen reader, was mind blowing.
PuTTY into Linux and you're in 16 colors.
My Linux terminal is 256 colors. Two hundred fifty six! That's like, every color!
24-bit color support (\033[38;2;rrr;ggg;bbbm) has been mainline in both Konsole and in libvte-based terminal emulators for many years. 16777216 colors. That's still not every color, but it's every color your monitor can display. When I wrote http://canonical.org/~kragen/sw/dev3/gradient.c in 02016 I had it in Konsole but not libvte, though I think it had been added to libvte upstream. http://canonical.org/~kragen/sw/dev3/gradient.png
Color in terminal emulators was one of the main perks of Linux over other Unixes for me at first!
I intentionally use 256 colors mode because I prefer the vibe :)
256 colors ought to be enough for anybody.
Laughs in Amiga
I’ve built systems for iSeries and none of the modern fancy GUI IDEs come close to the speed of those IBM 5250 terminals. You can still see such terminals in action in POCO baumarkt in Berlin.
One of the modules I saw in action was written before the moon landing, written by a lady programmer.
<16-colors IBM z/OS host terminal
This hasn't been virtualized?
From what I understand, they log in to a citrix-like webpage called IBM Host On Demand and then start a java applet which is essentially a 16-color 3270 emulator that's connected to z/OS mainframe.
They're probably using a 3270 emulator on a PC, but that emulator will still display 16 colors.
I am working as a mainframe developer at a bank, currently within a kind of data warehouse solution. Mainly writing in COBOL and Java (in USS), with some scripting in Rexx for internal ISPF tools. A lot of SQL as well, we use DB2 as our main database.
Most of our processes are EOD centric, we run a lot of batch jobs (mainly TSO, very little IMS). Integrations are mostly file based but we do both call and expose APIs (“regular REST APIs”) as well as consuming from and producing to Kafka among other things. We integrate with both mainframe and distributed systems on prem as well as “internal” systems hosted on cloud.
We use Git for source control but have a CI/CD solution that is built in house. Quite a lot of other infrastructure is also built in house.
I am mid 30s and am on the younger side looking at the mainframe area as a whole at my employer, however in my team we have several developers is in their 20/30s.
My background is mainly back- and frontend development on the Microsoft tech stack but I have worked, and do work, with whatever is at hand/required. But mostly stuff like .NET and SQL Server on the backend, and Angular/Vue/React on the front end before this.
What is crazy is how simpler is working with mainframes that host nation-wide apps, compared to developing a modern web app..
People joke about old coders brought out of their retirement to maintain a dusty COBOL/RPG program, while the reality is that the tooling is simple enough that a young developer could learn them in a month, and master in less than a year.
Plus, the expertise is not lost after a few years, given the platform focus on incremental improvement, and backwards compatibility.
What are wages like? I heard several years ago that you could land a sweet high paying COBOL gig, but from what I've read recently that doesn't seem to be true?
Met one close to retirement who worked on a ERP system in the food processing industry. Nightly batch jobs would trigger orders from their suppliers, customer service would enter new orders. Two SAP migrations already failed, costing the company millions. All company process knowledge was in code, database fields have been repurposed (but no renamed, too much work), feature development stop long time ago. In parallel a new system was built in-house (no longer trusting external consultants) and his job was explaining what the system does. Probably well paid but he didn't seem to care, he just wanted to work less and retire on good terms.
I quickly worked at a company like that, that had large parts of their core business logic running on an AS400, and they were asking what they need to do to migrate to something newer - was surreal. A few hundred employees all interacting with the system - in building A, a deliver arrives, they scan it like this, press this button. Building B, a guys job was to oversee the conversion of tables from As400 to a csv format suitable for some other outsourced software. Data goes into system C and shows to employees working on conveyor belts this and that. Hundreds of kilometers away, truck drivers get a notification for this and that.b
And, nobody knew how the whole worked. Everyone has their niche of interaction with the system. They would be able to shave off an insane percentage off expenses (in the form of employees whose job exists for no real reason), but the switching costs would also be immense.
I sometimes wonder what came of their company. The system was so far beyond the complexity that anyone could grasp, they had no inhouse devs, they'd need people with the competency to judge which competency they need.
I grew to like migration projects like that.
Currently working on migration of 30yo ERP without tests in Progress to Kotlin+PostgreSQL.
AI agents don’t care which code to read or convert into tests. They just need an automated feedback loop and some human oversight.
I would argue that they need heavy human oversight
For sure; I'll believe that an AI can read and "understand" code, extract meaning and requirements from it, but it won't be the same as a human that knows requirements.
Then again, a human won't know all requirements either; over time, requirements are literally encoded.
In systems like that you can record human interactions with the old version, replay against the new one and compare outcomes.
Is there a delta? Debug and add a unit test to capture the bug. Then fix and move to the next delta.
Not a COBOL developer, but working at a sizeable bank I witnessed the phasing out of their mainframes and AS400 systems. They ran some critical systems, both in retail and wholesale banking. They either converted to java, and optimized that code, but some COBOL code from the mainframe, and all of the AS400 stuff was converted into Micro Focus COBOL, which runs on Windows, which could be hosted on our Private Cloud. I worked on helping them migrate to our cloud infra, which was an interesting exercise. There was a very tangible cultural gap between the people maintaining and developing these applications and the rest of the organization.
Can you describe the cultural gap? I haven't really met these folks in the wild, so I'm curious what the programmers of yore were like.
In my experience, it's usually lack of awareness about modern security risks, and lack of familiarity with modern infrastructure paradigms. The latter really isn't a problem since these systems are usually standalone, but the former does become a problem - they often are from a time where this just wasn't something to consider. As a result, these legacy systems are often using default passwords, have tons of crazy stuff exposed to the network, and are comprised of custom code written specifically for the business purpose (so the documentation is only as good as what they made).
On the other hand, these guys generally write pretty neat, lean code that is quick, reliable, and directly responsive to the business. The really fun thing is watching the users fly through the keyboard-only screens, sometimes with muscle memory that is faster than the terminal emulator can update - they're literally working ahead of the screens.
Oh yes, I remember that when we swapped out a bunch of terminals at an airline.. The users complained it was all way too slow on the new Windows machines with MS SNA server in between... I was wondering what it was all about, as a young and very naive dropout from uni on his first IT job. When I came down, this dude was banging on his keyboard and after some time stopped, pointed at the screen and you could see it slowly catching up, screen by screen.. He showed me the directly connected version next. I learned something that day.
That's awesome. I set up Arch Linux a while ago, and despite working in Linux shops for more than a decade, let's just say I was very out of my element...
Reminds of me of a TUI Banking software that ran on Sun Solaris. It could keep up as fast as you can navigate - few months in and you could fly through the screens. Then it was "upgraded" to a web-based version and all of us were up in arms, it was like being downgraded to a tractor after experiencing a racecar.
Reminds of the DOS order management software I used in the 90’s.
ASCII tables, text only, with F key shortcuts. Hard to learn but blazing fast once you did.
Nothing modern approaches it.
As a support engineer at IBM we used a mainframe system called NRCPMA iirc... I think NR stood for Northern Region. Accessed via a terminal emulator, fully customizable with macros, fastest tool I ever worked with indeed, once you climbed the initial learning curve.
Reminds me of modern IDEs -- developers, both old and new, are too lazy to learn a complex IDE to speed up their work, even though it's their main tool for making money.
I don't think efficiency of navigating your IDE is a major factor in your productivity. If you like your setup, it's probably not going to matter a whole lot. The tool you make money with is your brain.
Hard disagree.
Few points I can easily remember:
1. Navigating the code, e.g easily see all the callers, navigate up/down the call tree requires static code analysis. Super handy while reading someone's else code, which is like 90% on large projects.
2. Quick refactorings. Often times I see people discuss in lengths what would/could be instead of just go and try it out quickly, seeing all the pros and cons. Many times I proven myself wrong by trying it out and seeing pitfalls I didn't see earlier.
3. Warnings: so many real bugs could've been prevented if developers had seen (or cared about) to IDE showing a warning. Many PR review suggestions are detectable by a proper IDE without wasting reviewer's time.
4. Hotkeys (what the parent comment was talking about) -- speeds up all of that, especially refactorings, freeing dev's brain for thinking of architecture and other problems.
I can go on an on. Sometimes it feels like 50%+ of AI usage for coding is to free up fingers, not knowing that they were already mostly free by using static analysis features/hotkeys.
It pays to help one's brain stay in the flow, though, and fast and reliable muscle-memory-based navigation of one's IDE does exactly that.
In my experience mainframes at financial institutions are hidden behind IBM middleboxes that are specifically designed to obviate the infrastructure risks. It's a classic example of a company selling you both the problem and solution.
That's just an example of incremental improvement. Mainframes and midranges adapt with the times without losing what works. Modern midranges, for example, can run C, Python, bash, and web servers.
I would say these people were in a relationship with the mainframe, if that makes sense. And also having worked at IBM in the past where I sat adjacent to the mainframe support team for Business Services, I totally get it. Mainframes are awesome if you ask me, and in a sense we have been trying to reinvent a lot of its goodness with "commodity" x86 hardware.
From a technical-cultural perspective it was mostly sulkiness, and a complete and utter lack of embracing the paradigms of distributed computing. Also, like most internal clouds, there were plenty of issues as it was. Initially they just tried to replace mainframe application components 1:1 onto VMs in whatever way and whenever anything was <100% reliable they complained that our cloud was not able to do it. I had to explain in a very harsh way, under a lot of pressure (I believe not hitting the deadline of switching off the mainframes meant renewal for a year at 40 Mil.. or thereabouts) the realities of "cloud".
The developers I spoke with in that time though, were very much the opposite of the move fast breaking things crowd. Intelligent, but also narrow minded I would say.
When I worked for a retailer whose logistics ran on IBM mainframes, one of the milestones was getting COBOL devs to use version control.
Glad to meet y'all. Yes. I code COBOL along with C, C++, Java, REXX, SQL, and IBM mainframe Assembler. I support the racehorse of IBM mainframe operating systems, z/TPF. There are few familiar with it. Think of z/OS as a Clydesdale horse, big and SLOW.
zTPF runs the airlines, the model for real online transaction systems... Not SQL relational database shopping carts.
Botton line, everything is Assembler. It's all just bits.
I've had a bunch of recent projects reverse-engineering old COBOL code, in financial services.
Mostly to figure out the best way to replace the old systems with something newer, so not really as a "COBOL dev", though.
Anything in particular you're replacing them with generally?
I heard a story about replacing COBOL with JavaScript ... and my skin still crawls thinking about it.
There's surprisingly a lot of finance related jobs in TypeScript. I wonder what libraries they are using for money management.
Heh I worked one of these. We handled arithmetic in the DB tho. Lot of PL/pgSQL running under the typescript, TS was more of like a middleware or API layer for things that could change more frequently. Finance code is to some extent transcribing of regulation & law into code and we kept all that in postgres.
Indeed, I've worked on billing system that relied heavily on pure JavaScript. Not even modern flavors with map/reduce, etc. - ECMAScript 5. It worked surprisingly well and our bottleneck wasn't the runtime but rather the databases we were constantly upserting to.
It sounds kinda crazy but with good change control, documentation, good relationship with the ETL team - it was pretty maintainable.
Any chance it uses Rhino?
Crossed fingers, in my experience.
But think of the flexibility, freed from the shackles of numerical coherence!
https://www.destroyallsoftware.com/talks/wat
I met a dev who's mom had been working on legacy banking systems her whole career. She had started in the eighties and she still did some urgent jobs at a crazy rate despite officially having retired.
My stepmom who retired five years ago, did COBOL dev as part of her banking job until 2002ish and then she was full-time management track. In her bank, most of the work had been integrated with Java, and the Java was done by outsourced Indian teams. At the time she retired she felt the Indian teams had been failing for years to meet objectives, and finally management was seeing it. Additionally everybody who knew the COBOL side of things was retiring at the same time as she was and she did not want to know what the system would look like in five years.
I can imagine the conversation involving something along the lines of “don’t ever call me.”
My mother used to teach Cobol back in the 80’s in Brazil but later she transitioned into management and haven’t touched a line of code for more than 30 years, she can’t even speak english wtf
Credo!
I haven't worked in COBOL, but I've worked with it.
This was around 1999, and I was building a system for configuring and ordering custom PC's at a large distribution company. One of the feature requirements was that we display inventory over the various options. (ie: There are 373 20G disks in stock, but only 12 30G disks in stock). The idea was that this would let a customer ordering 200 machines know that they should pick the 20G disk if the wanted it now.
The way inventory at this company was done was via two systems. There was a normal SQL database that had access to a daily snapshot of inventory taken from a mainframe that always had the up to date data. With the mainframe taking a while to process queries, we used the less current SQL database for the majority of the UI, but took the time to query the mainframe once a customer was in the shopping cart. Customers might see a change during the flow, but it would at least let them see the most current data prior to committing to a purchase.
The mainframe query itself was implemented by someone else as a COBOL job that produced the required inventory numbers. From my point of view, it was just a limited sort of query conducted over a specialized JDBC driver. (Not necessarily the weirdest aspect of that design.... for a variety of reasons, we did the UI in ASP/VBScript, the back end using Microsoft's JVM invoked via COM Automation, and the SQL database link was via a dubious use of a JDBC/ODBC bridge to connect to SQL Server. It all worked, but not the most solid architecture.)
==
My only other mainframe experience was working as an intern for a utility company a few years prior (1991-1992). They used CDC Cyber mainframes to run the power grid, using something like 4MM lines of FORTRAN code. The dispatchers themselves interfaced to the system using consoles with 4 19" color displays running at 1280x1024. Heady stuff for 1991. (The real time weather radar screen was cool too, in an age before the internet made it common place.)
Not Recently. 2010ish was working for an insurance processor that was actively writing thousands of lines and executing on z/OS with no plans to migrate.
I was part of team that was writing web applications that needed to call z/OS transactions. The IBM solution was to use their Transaction Gateway product, which cost a ton, and was slow as shit. We developed a framework that used annotations on the Java Side to map COBOL Records to Java Objects and invoke transactions over a TCP socket. Learning how to pack decimals and convert encodings was pretty cool. We ended up with a framework that was at least a zillion times faster than the IBM solution. Left that job though as the company was is distress and was losing customers (health plans). They eventually folded.
All I do is COBOL. Everyday. Our team has led over 40 “modernization” efforts, with about half migrating the apps to an X86 platform. There are “tons” of COBOL apps in Production, and not just limited to Financial organizations. I work with County, State and Federal systems. Retail, Insurance, Telecom, and Manufacturing. It’s not uncommon to work on programs that were original written in the 80’s or 90’s
The IBM Z is feature rich, but expensive and limited by software choices for apps not running ZLinux. (Think virtual Linux server)
The biggest catalyst now is the increasing license cost of mainframe third party software
I have written cobol in the past. I worked at a financial company that had a decent amount of cobol devs around, with the primary database being DB2. The code is mostly financial transactions and record updates, basically CRUD code. Cobol was essentially the backend with the front end being Java and Javascript/Angular.
My father is 75 and he still works, has his own software development company with his own back-office program written in cobol. He started a company back in 1991 with two other cobol programmers, they are retired now, and while almost all of the code they wrote has been replaced with c# code by younger programmers there are still some parts of the code written in cobol that he still maintains.
Random OT question: I was raised by, erm, relatively uneducated folks. Is there anything especially great about having a software programmer for a father? (As my kids have one)
They may eventually learn the valuable skill of politely smiling and nodding as someone talks with great passion about things with zero relevance to their day-to-day life, assuming they don't shut up about it at home.
Actually, tbf, my daughter's pretty interested in my coding and electronics projects. She picks things up alarmingly fast. I taught her a practically-one-way encoding scheme for passwords I've had incredible trouble teaching anyone else (LLMs also can't figure it out), and she completely understood it after the second example I gave and even added her own extra twist to it. Her passwords now are both memorable and extremely secure against dictionary attacks even with any mutation scheme someone could reasonably imagine.
-So I think that should probably go in the plus column.
Having your kids also be technical and bail you out of the shit you've built for yourself. Or at least nudge you away from bad ideas, do listen to them.
I'm being a little unkind to my Dad. He moved to management fairly early on and didn't really keep up with things.
He taught me a hell of a lot though, and did really know his shit at one point. It worries me how much his skills and understanding have declined over the years.
My brother works with COBOL for a bank here in Brazil, he is young (in his 20s), started before finishing his degree. Pay is poor, hours are insane, he is overworked as hell, and anything “modern”, like git, is out of the question.
He’s trying to learn Go now and modernize himself to see if he can get out. I’m trying to help as much as I can. Hopefully, he’ll land a job somewhere else this year.
I can't understand why a financial institution would underpay someone who has such responsability.
The recent hacking of BMP shows the risk this creates (poorly paid employee with debts sold his password to hackers).
at the banking place I work running things in k8s, z/OS-people are actually the ones running custom git clients in go on z/OS. Bonus: they have no nosy Java devs (recurringly producing threading bugs...) saying "but we all use spring boot!!!" and likely no manager asking "is this cloud-ready???"
What I'm wondering: Are the salaries high? Not just because you've been employed at the job for a long time with regular raises, but because it's hard to find developers.
No the salaries aren't high. They are typically lower than other software engineer salaries. There are a large number of contractors from Indian consulting companies with "experience in cobol" to make run of the mill cobol cheap enough.
The very high salaries you hear about sometimes are always for VERY specific mainframes that are extremely old with a lot of quirks, and are usually being paid to consultants.
> There are a large number of contractors from Indian consulting companies with "experience in cobol" to make run of the mill cobol cheap enough.
Seeing the horrible performance from Indian offshore firms with modern languages I cannot imagine the mess they make with legacy languages like Cobol. Or is it the other way around?
Corps operate despite inefficiency not because of efficiency. LLC protection and market control are everything.
But the code still has to work. LLC's and other corporate structures only protect the owners if the company goes bankrupt, which it will if its systems stop working.
Ditto with market control, it's not some permanent crown you achieve. Companies have to keep performing to keep their market share.
E.g., if you opened an account at a major bank, and your transactions started failing, would you keep banking there?
> E.g., if you opened an account at a major bank, and your transactions started failing, would you keep banking there?
A lot of people who land in that situation do continue banking there since they are either tied into that bank through loans/debt, or lack the time/energy to move elsewhere.
This situation would lead to immediate and extremely severe legal and commercial consequences for the bank, even if it's JP Morgan.
The argument that a market leader can screw up because it "owns" the market is not correct. Where's Xerox / IBM / Intel now?
A while back I came across job listings for a COBOL consultancy near me that only seems to hire fresh grads for well below market rate (not much higher than retail/restaurant jobs - this is in a cheaper part of the US). They promised to train their employees from the ground up and implied that COBOL knowledge would set them up for a really profitable career. It seems like they were taking advantage of the common advice: "just become a COBOL developer, it pays well because nobody wants to use COBOL!" But I'm skeptical that someone coming out of that consultancy with 2 or 3 years of experience in nothing but COBOL would do well on the job market.
The places I know that use (or used cobol 5 years ago) were all in hiring freezes for cobol developers and were trying to get off of it as much as they could (no new development, only maintenance, etc). I don't think its a surefire bet.
I suppose it varies by country?
In my country, no, COBOL jobs aren't well paid. They are below average.
Global Shop ERP is written in Visual Cobol, believe it or not. Supposedly they are actively rewriting the eight million lines of code to C#.
Related [0]
[0] https://research.ibm.com/blog/watsonx-code-assistant-for-z-i...
I have a friend who works writing COBOL for a US state government, I sent him this post and this was his response:
I work in COBOL for government systems.
We upgraded our old mainframes to IBM Z16's, and we just got some new ones in recently for our backup server.
Part of the job is taking whatever they decide in legislation and translating that into code that processes whatever they decided to make law, from fees, suspensions, special legislation for certain areas, etc.
Our programming environment primarily uses TSO/ISPF and changeman for version control. We have access to IBM's IDz (previously RDz) as an IDE for development, and I would personally prefer to use that but haven't been able to get it installed on my work computer due to licensing issues. Part of security protocol is that we cannot use anything open-source, so VSCode with Zowe is, unfortunately, out of the picture.
We maintain a lot of old programs and modules, but we are also actively developing new programs as we expand our IT department - and yes, that is new COBOL programs. We have a Linux side as well which mainly deals with the web-side, but they still interact with the mainframe to call on modules - but all they are really doing is sending data to CICS to get data back. They do not know anything about the COBOL itself or how to program in it.
In undergrad(around 2005) we used to have a COBOL lab, most of the assignments were pretty much add/update/delete types. A friend of mine found COBOL so boring, in order to keep it interesting he wrote COBOL program for brothel(not real one), to maintain the customers, prostitutes, transactions and whole nine yard. Despite of not liking the language, he was the best COBOL programmer in college.
Any active COBOL devs here?
A legitimate question, but so far not many answers, and they're mostly from people who know people who know COBOL devs. This is to be expected.
Demographically, COBOL devs skew older, and there aren't a lot of graybeards left on HN. This place used to be full of them, and they always had interesting and unusual insights and techniques to share. Those days are long gone.
IMO, Graybeards have largely left HN for a few reasons:
- They're tired of being shouted down by the Reddit-quality ageism that lingers through this forum.
- They're mature enough to no longer be interested in chasing every little tech fad as if their lives depended on it, and that's 90% of what HN has become.
- As most older people do, have other things in their lives that are more interesting than work. Family. Children. Hobbies. Traveling. Service. The world is full of things more rewarding than being terminally online, or being reminded of your day job.
I applaud your curiosity, but you're standing in a church asking, "Where are all the atheists?" COBOL devs aren't here. And where they are is likely not online.
> IMO, Graybeards have largely left HN for a few reasons:
If such an online community exists, where did these graybeards go to?
The tinkers are on vcfed. All of the others I know (not many, maybe a dozen) are barely online outside of work. They're largely bored with the internet.
>...they always had interesting and unusual insights and techniques to share.
We need more like this, please.
This is in my past, so no COBOL recently but in one of my first consulting gigs I wrote several business programs in RPG III. Which is the nightmare that you might imagine it is. Think plugboards. Halfway through the IBM guy came by and installed COBOL and wow what a difference. My last act was to recommend that they get a computer department. And so Tom Monaghan called IBM
Apropos of nothing in particular, there's an older HN thread about "Good resources for learning COBOL" that some folks here might find interesting. OK, calling it a "thread" is over-stating things, but still...
https://news.ycombinator.com/item?id=18479536
Actively working on multiple COBOL systems in the Healthcare field, under the umbrella of “Patient Accounting”. Still developing new programs and making modifications. I’d say these applications will be active for at least another 5 years!
For my sins, I work in Education supporting PeopleSoft. That means that I do work in COBOL on occasion. I am not tasked with writing anything new in COBOL, but I so quite a bit of analysis and support. That is: I _read_ COBOL more than I write it.
There are three flavors of COBOL that I deal with: PeopleSoft delivered, Vendor delivered, and University modified. Most of the work I do in COBOL breaks down to reading the code to determine why a given behavior is observed. Only once (in University modified code) have I needed to make an actual edit. The rest of the times I either modify the flow of information into the COBOL code or I modify the resultant after the code has run.
Worked at a large insurance company in the late 90s where leadership touted a big benefit of converting to PeopleSoft Financials as part of their Y2K remediation was the elimination of their dependency on the archaic COBOL language, blissfully unaware that their new PeopleSoft applications were using Micro Focus COBOL under the covers. Oops! https://blogs.oracle.com/peoplesoft/post/take-note-significa...
Going on thirty years ago, I had some experience with Peoplesoft. I'm interested to hear that it still uses COBOL, though on the other hand, why shouldn't it?
I am not a COBOL dev but have worked on a few codes here and there. I played around with LLM and it appears to be not too bad at COBOL, what’s the consensus or your experience utilizing LLM for COBOL?
I worked as a cobol dev in my first fulltime job. I did mostly cobol85 (microfocus) and object-cobol (netexpress/fujitsu cobol), with isam files. Did the whole "migrate to year 2000" process in a couple of big but old applications (healthcare management software and manufacturing management software). Cobol is a school by itself; there are so many ways you can shoot yourself in the foot that one either learn how to structure programs avoid a spaguetti mess, or slowly descend into madness trying to fix bugs :)
Government. About 25% of my job. No day to day mainframe development, but we do need to update some logic for new policies and regulations.
There’s not much maintenance work. There are very few bugs, as the core applications have been running for decades, most come up with interactions to external services.
Any major development projects are only in service of lower overall COBOL development time, like transitioning some business logic to database updates.
And there is a decommission plan for the mainframe, so plenty of work helping that team.
I know some COBOL devs. They work in a bank.
They wouldn't hang around here though.
Where would they hang around? Curious to learn about other programming/tech communities
I worked in an IBM mainframe shop as my first job. The COBOL devs for the most part did not participate in programming/tech communities. It was a 9-5 job and they didn't think about it otherwise. Maybe they would have a subscription to Datamation magazine or something like that but I doubt it.
But keep in mind there was no internet, no podcasts, no youtube. If you needed to learn something you learned it at work, on work time. If something new was introduced, IBM came in and conducted training, at work. For something big you might get sent to an IBM training center for a week. There was no need for (and no real way to do) any learning on your own outside of work.
They aren't really into tech outside of work.
Same, lately.
... or, arguably, inside of work. :)
There's a somewhat active community of COBOL developers here:
https://www.reddit.com/r/cobol/
If you're every really bored, search around the HN archives to find out how I accidentally founded that community as a result of a joke. :-)
I do other things besides COBOL but I maintain and occasionally enhance many COBOL applications:
- Item Database (SKU, UPC, attributes)
- Stock Status (Sales, Onhands, process sales transactions from POS systems)
- Replenishment (Create vendor and DC orders to stores)
- Reporting (Report sales and other performance data by stores, departments, classes, etc..)
- Pricing (make and maintain current and future price changes, send to store when needed).
- Many other applications (15+)
They have been saying they are going to get rid of these applications for over 20 years now. I am still waiting...
Learned COBOL in the 90s as an undergraduate at university. Took it at the same time as PC Assembly language (woefully 16 bit).
I had less trouble with assembly than COBOL at the time I'm afraid.
They're both weird beasts, but once I learned the C calling conventions for assembly, I was able to make a lot more sense of the assembly.
COBOL is a world unto itself. I didn't hate it, but I didn't think I saw a career in it either.
I'm just glad I opted not to take RPG :-)
I don't code in Cobol myself anymore but I manage team who does.
We work on Phoenix, government of Canada payroll system. If you google it up, you'll see some interesting coverage. However, the underlying peoplesoft ERP itself is rock solid at every other client I've served over last 25 years. Peoplesoft uses Cobol and sqr, as well as proprietary languages stored in database, application engine and peoplecode.
Key payroll processes are in Cobol. This is because of its tight integration with database and ability to manually control database cursors. We are very database oriented when it comes to performance. Our developers need to know the programming language, but also deep understanding of client business processes, and sql optimization. They also work closely with our dbas to ensure good performance. So our developers are technically proficient in Cobol and couple of other languages, but also very very strong in sql optimization, and understand clients payroll rules and can speak intelligently with compensation advisers and payroll processors.
I personally found that to be true for most Cobol programmers - whereas typical hacker news Dev seems very technology oriented and frequently moving, typical Cobol programmer is very business process aware and integrated with corporate line of business. They don't move as much for several reasons, but that deep awareness of client is one of them.
Edit: I shoild mention, while peoplesoft can and does work on mainframe, most of my clients are on windows, Linux, or AIX. COBOL is not quite as mainframe specific as it sometimes seems :-). See e.g.microfocus Cobol for a modern multi platform compiler.
Not COBOL but years ago I did some work on IBM AS/400 using RPG. For those who don’t know, RPG was originally written (in 1959) for punch cards and the programs had to be written in a pattern that resembles punchcard. It was a fun experience and I am grateful to have dabbled in it.
Many megacorps still run AS/400 and it’s uptimes and performance is legendary.
Edit: Forgot to mention that I was mentored by folks more than twice my age that time.
Not a COBOL dev myself but for many years I worked in government healthcare for my state. Probably 75% of developers and BAs for the organization were working to support an extremely large mainframe system(s). I worked as a web developer and we had to use a product called Hostbridge that allowed our web applications to interface with the mainframe via javascript.
I worked with COBOL in banking, a lifetime ago. It was one of my first jobs.
Batch jobs, clunky and very verbose programs, nothing interesting. I... hated it.
About 10 years ago I was working on a net-new COBOL compiler for z/OS. There was a huge demand from banks especially for this.
I've done COBOL since the 80s, but I don't have a paying gig for it anymore. That said, I have situations where I need to process a flat file of data, and I find it a lot easier to just fire up gnucobol and write the program than try to remember the syntax for python or golang.
Followup question: what’s it like to write COBOL? It’s so ancient as this point you never hear anyone actually talking about what the developer experience is like. How performant is it? What features does it have (or not have)? How bad is it really?
I knew a guy who wrote a lot of 360 assembler back in the day, never a COBOL programmer.
I was a consultant at a very large insurance company in the 90s when the dictate came from the top “no more assembler”. There were a few groans from the audience of developers.
I still remember this crazy time where COBOL engineers were in dire need: https://news.ycombinator.com/item?id=22787828
2075 HN: Ask HN: Any active Java devs here? What LLM do you use?
C++77 is coming soon, better study what's coming.
In 2075 that will be like asking blacksmiths what kind of anvil they use.
Not me, but not even 5 years ago one of my friends was working on Cobol codebase running on IBM zOS. It did not last long, but as far as know they had a decent time.
A note-taking app.
Maybe the COBOL devs aren’t here.
Ex COBOL dev here. Started at NCR on COBOL for minis and mainframes. Then built stuff on Wang VS COBOL for ERP systems until 93. Quite a few other languages since then but been a Product Manager for past 20+ years.
Same here, I worked on Wang VS COBOL too. IMO, it had the best TUI Screen generation Tools I have ever seen.
Towards the end, I worked on a project to port Wang VS OS and its COBOL to AIX. I was tasked finding issues with COBOL Programs we had on the VS. It was a good environment, but Wang went CPT 11 before it was ready :) It was rather close to being complete.
I worked on Wang VS COBOL too, great environment. Also worked on Wang PACE, their 4GL/RDBMS, the people who developed that stuff did a great job.
https://prog21.dadgum.com/143.html
HN is a place where everyone is, so it's reasonable to assume that some Cobol devs may also lurk here.
I am here; so I can attest to HN being a place for everyone.
Perl devs too. There are literally dozens of us!
> There are literally dozens of us!
I am happy that you are here but there is no need for hyperbole.
Everyone except most developers I know.
This is oddly true in my experience. I ask people where they get their information from and the channels are generally a lot more sparse than topics posted to HN.
If I were to venture to guess: I'd say "most developers" seek work-life balance and aren't interested in reading long-form articles on how to do X or Y when they are off the clock.
Shout out to all the MUMPS programmers out there reading HN. May your code always be healthy.
There's more MUMPS programming going around than many people believe, especially in financials...
Including new development.
I’m not affiliated, but this made me think of this AI for COBOL startup: https://www.cobolcopilot.com/
For some reason I think we’re all drawn to the idea of working with an older language. I wonder why!
Y'all commented about the number of terminal colors. That's funny. Only 1 color is needed for programming. The IBM mainframe terminals (3278) were called Green screens for a reason.
Work on a dual Java/COBOL team in a F500. Not a COBOL dev myself but I'm surrounded by them. Two biggest problems are:
a) the systems are very tightly coupled, like uber-monolith architecture, and it's hard to QA them without end-to-end testing through everything. Good luck getting anyone to refactor them because a1) they're going away and a2) they're so huge. Which leads into . . .
b) there's 40 years of undocumented business logic in there, or business logic that only exists in code comments. And it still makes billions of dollars a year. So good luck replicating 40 years of tribal knowledge in the new systems.
c) It was written by cowboy coders 40 years ago when the company was a startup, so no one can learn to work on it without first getting hired here. The joke is one of the original architects went and created his own dialect of COBOL.
not cobol but we do a hell of a lot of business basic.
Cloud migrations
Rewriting things in Java? Or maybe running Cobol in the cloud directly? A mainframe has a number of properties of the cloud, in a sense.
Bank.
Can LLMs do Cobol?
> Can LLMs do Cobol?
I imagine it's the one place where LLMs would absolutely shine. COBOL jobs are usually very verbose, lots of boilerplate, but what they do is mostly very straightforward batch processing. It's ripe for automation with LLMs.
The flip side is that banks are usually very conservative about technology (for good reason).
I don't think LLMs are anything human language specific, so would they really shine here? I.e, COBOL and SQL may be great for humans who otherwise aren't used to programming languages, but LLMs have seen everything, and thus are able to know any (programming) language, not just ones which are English-like.
I think it would shine because COBOL is very verbose and the programs written with it (batch jobs) very straightforward but also very boilerplate-rish and boring to write by hand, a situation with little risk and high reward to automate.
IMO the ideal path to working on COBOL without having decades of experience would be to spend a few days groking the syntax and writing toy programs for practice, then collaborate with a large LLM to understand the current code and gradually make changes.
Have you worked with COBOL? My understanding is that the language itself isn't the real problem. Mainframes, control languages: everything around COBOL is very different from most Windows/UNIX people have experienced.
Yep. I have recently used prompts to ask for COBOL code solution so I can compare it to other languages I also know to check quality of the answer. So far no mistakes.
I'm sure they can do brainfuck if you have a good training set.
LLMs are terrible at brainfuck. I spent a solid week attempting to use generative models to iteratively refine BF program tapes with nothing to show for it. I've written genetic programming routines that can produce better brainfuck programs than ChatGPT can.
For example, if I prompt ChatGPT: "Write me a BF program that produces the alphabet, but inverts the position of J & K" it will deterministically fail. I've never even seen one that produces the alphabet the normal way. I can run a GP algorithm over an example of the altered alphabet string and use simple MSE to get it to evolve a BF program that actually emits the expected output.
The BPE tokenizer seems like a big part of the problem when considering the byte-per-instruction model, but fundamentally I don't think there is a happy path even if we didn't need to tokenize the corpus. The expressiveness of the language is virtually non-existent. Namespaces, type names, member names, attributes, etc., are a huge part of what allows for a LLM to lock on to the desired outcome. Getting even one byte wrong is catastrophic for the program's meaning. You can get a lot of bytes wrong in C/C++/C#/Java/Go/etc. (e.g. member names) and still have the function do exactly the same thing.
That was a very serious response to an off-the-cuff joke.
BUT: Please, oh please, write up a blog entry! I bet that would be fun to read.
Yes! But only if they write the compiler too.