I don't normally leave feedback on these posts, so I apologise if I get the tone wrong and this is overly harsh.
1. If you're going to record a demo, invest $100 into a real microphone. The sound quality of the loom demo is really off-putting. It might also be over-compression, but it gives me a headache to listen to this kind of sound.
2. The demo has left me more confused. Rather than going step by step, you take the Blue-peter approach of "Here's one I made earlier" and suddenly you're switching tab to something different. Show me the product in action.
I guess I'm not in the market for this, but it feels like UI-heavy for something that's evaluating agents / infrastructure-as-code. I'd have thought if I was going to not just automat something, but also automate the evaluation of that automation, then I'd want a pipeline / process for that, and not actually scan down the criteria trying to work out which blog-posts are which and how the scores relate.
Thanks for your feedback on the video. Great point about going step by step, instead of switching mid-stream to a pre-built session :). We have another even simpler version which goes slower, step by step which would have been better for this post? The challenge has been balancing between showcasing the wide feature set with duration.
We have a spreadsheet integration (which I might post as a comment) for the usecase you mentioned. The scorer is quite light weight so easy to integrate it in your existing pipelines instead of building yet another pipeline/framework. The co-pilot is specifically for triangulating the right set of metrics (that are subjective based on your taste), which does require looking at examples a few at a time and make a judgement call. But I agree that once you are done with that you want to quickly transition off of this to either code or other frameworks like sheets, promptfoo etc.
Hey HN: I'm John, an engineer from Pi Labs. Despite lurking on HN daily for over 10 years, I've never posted, and now feels like a good time to change that :)
I joined Pi 3 months ago after a decade at Google. It was partly the HN community that inspired me to make the switch to a smaller company where I could have more direct impact. Working at a start-up has been quite an adjustment: while the work is extremely rewarding and fun, the pre- product/market fit phase is challenging in ways I've never experienced before in my career.
That's why I asked the team to post here, and am excited to show off this launch to see whether it meets a need that developers have (or learn why if not!)
Just for fun, I spent the last 5 minutes making an evaluation system for Show HN posts, so you can look at a real example if you'd rather not make your own [1]. If you sign in, you can fork and modify it, but you can also go directly to the homepage to try your own hand at it without any sign-in.
There was a thoughtful reply here with some feedback on the scores. It seems to have been deleted while I was writing a reply. In the interest of substantive discussion, I'm posting my reply below, since I think it's still valuable information.
--
Yes, I noticed there were a few things off about this example (e.g., certain questions feel inapplicable for certain examples, or certain scores feel too optimistic for the bad example), and I intentionally left them there so as not to window dress things too much.
To add to some of your observations, I'll note that:
1. Automatic question generation runs before I've given any examples in this particular chat. This can be a positive (in that you can get started without even providing an example of your own data), but it also means we sometimes add questions that don't make sense for the data you actually have. The co-pilot is meant to be iterative for that reason. (As an example, towards the end of the chat, I do ask it to remove some questions that don't feel applicable).
2. The model still has to output a score for all questions, even if they don't apply to a particular input. We're working on a new system that will understand which questions actually apply, and can turn certain questions off if they're impossible to answer given the inputs provided.
3. We do get feedback from users that the scores feel off sometimes. In some cases, they're too high; in others, they're too low. We're working on an interface for calibrating the scores to your own preferences, e.g. with a small amount of thumbs up/thumbs down data. There's a tradeoff here, though, because we're also trying to make the evaluation process a lot easier than today's "prompt an LLM as a judge" paradigm where writing a prompt with a rubric can take a substantial amount of time, and any kind of calibration adds friction for users.
Overall, we release new models every other week and track ourselves on internal benchmarks to see improvements to both question generation and question scoring. If you play around with the system more and find other ways it's not working as you'd expect or like, please feel free to email me your examples and we'd be happy to prioritize looking into them.
I don't normally leave feedback on these posts, so I apologise if I get the tone wrong and this is overly harsh.
1. If you're going to record a demo, invest $100 into a real microphone. The sound quality of the loom demo is really off-putting. It might also be over-compression, but it gives me a headache to listen to this kind of sound.
2. The demo has left me more confused. Rather than going step by step, you take the Blue-peter approach of "Here's one I made earlier" and suddenly you're switching tab to something different. Show me the product in action.
I guess I'm not in the market for this, but it feels like UI-heavy for something that's evaluating agents / infrastructure-as-code. I'd have thought if I was going to not just automat something, but also automate the evaluation of that automation, then I'd want a pipeline / process for that, and not actually scan down the criteria trying to work out which blog-posts are which and how the scores relate.
Thanks for your feedback on the video. Great point about going step by step, instead of switching mid-stream to a pre-built session :). We have another even simpler version which goes slower, step by step which would have been better for this post? The challenge has been balancing between showcasing the wide feature set with duration.
We have a spreadsheet integration (which I might post as a comment) for the usecase you mentioned. The scorer is quite light weight so easy to integrate it in your existing pipelines instead of building yet another pipeline/framework. The co-pilot is specifically for triangulating the right set of metrics (that are subjective based on your taste), which does require looking at examples a few at a time and make a judgement call. But I agree that once you are done with that you want to quickly transition off of this to either code or other frameworks like sheets, promptfoo etc.
Hey HN: I'm John, an engineer from Pi Labs. Despite lurking on HN daily for over 10 years, I've never posted, and now feels like a good time to change that :)
I joined Pi 3 months ago after a decade at Google. It was partly the HN community that inspired me to make the switch to a smaller company where I could have more direct impact. Working at a start-up has been quite an adjustment: while the work is extremely rewarding and fun, the pre- product/market fit phase is challenging in ways I've never experienced before in my career.
That's why I asked the team to post here, and am excited to show off this launch to see whether it meets a need that developers have (or learn why if not!)
Just for fun, I spent the last 5 minutes making an evaluation system for Show HN posts, so you can look at a real example if you'd rather not make your own [1]. If you sign in, you can fork and modify it, but you can also go directly to the homepage to try your own hand at it without any sign-in.
[1] https://withpi.ai/project/Xxyhrg2UR8kZHeNmbdV3
There was a thoughtful reply here with some feedback on the scores. It seems to have been deleted while I was writing a reply. In the interest of substantive discussion, I'm posting my reply below, since I think it's still valuable information.
--
Yes, I noticed there were a few things off about this example (e.g., certain questions feel inapplicable for certain examples, or certain scores feel too optimistic for the bad example), and I intentionally left them there so as not to window dress things too much.
To add to some of your observations, I'll note that: 1. Automatic question generation runs before I've given any examples in this particular chat. This can be a positive (in that you can get started without even providing an example of your own data), but it also means we sometimes add questions that don't make sense for the data you actually have. The co-pilot is meant to be iterative for that reason. (As an example, towards the end of the chat, I do ask it to remove some questions that don't feel applicable).
2. The model still has to output a score for all questions, even if they don't apply to a particular input. We're working on a new system that will understand which questions actually apply, and can turn certain questions off if they're impossible to answer given the inputs provided.
3. We do get feedback from users that the scores feel off sometimes. In some cases, they're too high; in others, they're too low. We're working on an interface for calibrating the scores to your own preferences, e.g. with a small amount of thumbs up/thumbs down data. There's a tradeoff here, though, because we're also trying to make the evaluation process a lot easier than today's "prompt an LLM as a judge" paradigm where writing a prompt with a rubric can take a substantial amount of time, and any kind of calibration adds friction for users.
Overall, we release new models every other week and track ourselves on internal benchmarks to see improvements to both question generation and question scoring. If you play around with the system more and find other ways it's not working as you'd expect or like, please feel free to email me your examples and we'd be happy to prioritize looking into them.
I think naming things co-pilot is getting excessive. Its just diluting your potential brand with MSFT and the term itself seems to be meaningless now.
While MSFT may not own co-pilot, they definitely control the mindshare.
See also: https://news.ycombinator.com/item?id=42781316