OpenAI says its latest models make significant strides in assisting work on low-resource programming languages, a class of languages that includes COBOL, Fortran and Visual Basic and that underpins critical legacy infrastructure. The company published two targeted evaluations — MultiLangBench and a COBOL migration benchmark — that measure whether models can build new systems across many unusual languages and preserve behavior when migrating legacy code, and it reports material improvements in accuracy and API cost for its newest models.
MultiLangBench is designed to test whether an LLM can implement a SQLite-compatible database engine from scratch across 20 programming languages. The benchmark requires models to organize a codebase, manage persistent state, implement SQL behavior and preserve existing functionality while adding features. To make results comparable across languages, candidate programs operate on a provided database file and must produce the expected modified database after executing a SQL query; OpenAI adapted tests from SQLite’s test suite and reports both test pass rate and estimated API cost over extended coding sessions.
On MultiLangBench, OpenAI reports large gains for its newest models. At maximum reasoning effort, GPT-6 Astra reached an average test pass rate of 87.4% across the 20 languages, up from 26.8% for GPT-5.6 Sol. Astra improved in every language tested, with especially large jumps in COBOL (from 10.9% to 72.1%), OCaml (25.6% to 85.8%), Fortran (21.4% to 83.5%) and Visual Basic (27.5% to 89.4%). Astra scored above 83% in 19 of the 20 languages; COBOL was the most challenging at 72.1%.
OpenAI also presents GPT-6 Luna as a lower-cost alternative that nonetheless improves on GPT-5.6 Sol in every language. At maximum reasoning effort Luna achieved a 40.3% average pass rate. Language-level gains for Luna included COBOL (10.9% to 17.1%), OCaml (25.6% to 34.1%), Fortran (21.4% to 31.4%) and Visual Basic (27.5% to 41.6%). OpenAI estimates Luna’s average API cost per sample at $0.77 versus $3.48 for GPT-5.6 Sol, roughly a 78% reduction in cost.
OpenAI has published the MultiLangBench evaluation setup and test suite on GitHub so teams can inspect the tests, view per-language results and measure gaps relevant to their codebases: https://github.com/openai/MultiLangBench.
The company’s second evaluation focuses specifically on migrating COBOL programs to modern languages — C++, Java, Kotlin, Python or Rust — while preserving observable behavior. Accurate migration requires understanding COBOL’s semantics and business logic embedded in legacy code, including fixed-point arithmetic and record-handling conventions that can materially affect outputs. OpenAI evaluates migrations with hidden tests that require byte-for-byte parity on observable outputs; a migration is considered successful only if all tests pass.
On the COBOL migration benchmark, OpenAI reports dramatic improvements. GPT-6.1 Sol achieved 86.6% accuracy at low reasoning effort, compared with 5.4% for GPT-5.6 Sol at the same setting. OpenAI notes GPT-6.1 Sol delivered these gains while costing 39% less than GPT-5.6 Sol (i.e., about 61% of the prior cost). At higher reasoning effort, GPT-6.1 Sol’s accuracy reached 97.9%, bringing the benchmark close to saturation.
Taken together, the MultiLangBench and COBOL migration results indicate the new models are both more capable across a wider range of languages and more cost-efficient for extended coding tasks. For enterprises that maintain legacy systems, the improvements are directly relevant: better model support for low-resource programming languages can assist maintenance, modernization and behavior-preserving migrations where business rules and numerical behavior are encoded in old code.
OpenAI frames these evaluations as instruments for tracking progress and identifying remaining weaknesses. The company says developer and enterprise feedback guided the work, noting common maintenance challenges such as missing documentation, departed original authors and business rules expressed only in legacy code. OpenAI also cautions that, despite strong benchmark performance, model-generated migrations or code intended for production systems still require expert review and full testing against an organization’s business rules and dependencies.
The latest models are available through OpenAI’s API for coding tasks, and the company invites developer feedback on strengths, weaknesses and priority languages or workflows. Teams can reproduce the MultiLangBench tests, compare model performance in the languages they use and report gaps that matter to their codebases. As support for low-resource programming languages improves, organizations working on legacy systems will have new tools to accelerate maintenance and modernization — provided they combine model outputs with careful engineering review.
In short, OpenAI’s published evaluations show notable progress on low-resource programming languages, with higher pass rates and lower costs on both cross-language implementation and behavior-preserving COBOL migrations. The results offer a practical benchmark for organizations assessing LLM-assisted modernization workflows while underscoring the need for expert oversight before deploying generated code in production.
Source: Read the original source
Leave a Reply