What are the limitations are there in terms of tasks this can handle? How does this compare with the other products out there? There are plenty of options...
Depends on your set of tasks but we use Engine for the bottom ~50% of issues by complexity. We have a pretty good swe-bench score from a while back but it's got much better since!
We have also focused on workflow integrations so you can assign issues from Linear, Jira, Trello etc which makes it more useful for teams.
Seems to me that integrations will be the most important component of tools like this. As an engineer I get my context from video calls with customers and other engineers, slack messages, emails, docs online, using the product myself, etc. So an auto-engineer should do the same.
This is cool. I can see the anti-monopoly of OpenAI argument, but apart from that is there a strong argument of being multi-LLM for a Codex-like agent?
We often find that some models perform better on certain types of repo. For example Claude 3.5/7 is typically much better at frontends. That's why we let you switch up the model for each repo.
We last submitted a SWE-Bench verified result in November 2024 - at the time I believe we were in the top 5 entrants.
We expect Engine to be as good as the other code-writing agents out there at the moment - we understand almost everyone in the space to be using very similar base models and agent scaffolding.
I worked on this! Happy to answer any questions anyone has.
What are the limitations are there in terms of tasks this can handle? How does this compare with the other products out there? There are plenty of options...
Depends on your set of tasks but we use Engine for the bottom ~50% of issues by complexity. We have a pretty good swe-bench score from a while back but it's got much better since!
We have also focused on workflow integrations so you can assign issues from Linear, Jira, Trello etc which makes it more useful for teams.
Seems to me that integrations will be the most important component of tools like this. As an engineer I get my context from video calls with customers and other engineers, slack messages, emails, docs online, using the product myself, etc. So an auto-engineer should do the same.
This is cool. I can see the anti-monopoly of OpenAI argument, but apart from that is there a strong argument of being multi-LLM for a Codex-like agent?
We often find that some models perform better on certain types of repo. For example Claude 3.5/7 is typically much better at frontends. That's why we let you switch up the model for each repo.
I've already merged my first Engine PR! Being able to review PRs like normal and it updates its work is very cool.
Looks great! What's your experience of using this for working on real world production code?
60% of the time, it works every time
Looks cool - do you have any benchmarks? How do you compare to other products out there?
We last submitted a SWE-Bench verified result in November 2024 - at the time I believe we were in the top 5 entrants.
We expect Engine to be as good as the other code-writing agents out there at the moment - we understand almost everyone in the space to be using very similar base models and agent scaffolding.
Demo’s this 6 months ago. Super excited to see how far it has come since!!!
been following engine from afar for a while, super cool to see it on HN. didn't see it had a free plan, will try it out.
[dead]
Boom been waiting for something like this!