I'm one of their customers. I often see that one green car parked down the road.
It's pretty good - their provided router is locked down to hell and they're on a cgnat, but not having to deal with Comcast's 1.2tb data cap is well worth it. Checking Comcast's site now, it seems that they now offer "unlimited" data. Interesting, that option wasn't there 6 months ago.
~100 customers seems too small for the amount of effort they have put in so far. They've been working along all the roads near me for about a year, and they're out there running fiber conduit every day. The houses out here are far apart. Hopefully, they can make it work.
> I'm one of their customers. It's pretty good - their provided router is locked down to hell and they're on a cgnat
This sounds like mine. I'm guessing yours doesn't support IPv6 because most fiber providers don't.
For the router, I already build firewalls so that. I pay $10/mo to escape their cgnat.
I've also alerted them to expect regular haranguing from me about deploying IPv6. Especially since bgp.he.net shows they have a /40 allocated to themselves; it doesn't seem to be used.
I've had less than 0.5% of customers ask for IPv6 from my fibre ISP. It's not worth supporting as a result. The main reason is that any service that is not widely used will have gremlins that result in poor customer experience, and if it's always the same handful of customers hitting problems or finding quirks, there is a real risk of poor word of mouth incident reporting that can harm the business. At least if something goes wrong with IPv4, it's going to be noticed very quickly.
Some people will say monitoring is all that you need, but I do not agree. There are a million different little issues that can and do occur on physical networks in the real world, and there's no way monitoring will have a 99% chance of detecting all of them. When incidents like the partial Microsoft network outage that hit certain peering points occurred, I had to route around the damage by tweaking route filtering on the core routers to prefer a transit connection that worked over the lower cost peering point. It's that kind of oddball issue that active users catch and report which does not happen for barely used services like IPv6.
> I've had less than 0.5% of customers ask for IPv6 from my fibre ISP
How many ask for IPv4? I understand your situation, it's a lot of work, for something that many won't notice. It's just that saying there's no demand because your average consumer, who also doesn't know what IPv4 is, isn't asking for it, is the mentality that keeps IPv6 from being implemented.
On the funnier side of things, we've also sometimes run into the opposite problem that we can't reproduce an issue, because it's only on IPv4 and 95% of the time everything we do is IPv6. But we're also not serving home users.
Static IPv4 addresses are closer to around 5% of customers. Nobody asks for IPv4, but some customers bring their existing or own wireless routers along and occasionally choose devices that are not IPv6 capable. Maybe in another 10 years those devices will finally be fully removed from service. The worst stragglers right now are the old combo DSL modems that effectively have no modern replacements -- it's just not worth spending money to replace them when customers are going to migrate to fibre soon enough.
I don't think GP meant static addressing but literally, how many ask for IPv4 service? None, because you just provide it; it's an unstated expectation.
Now apply that to IPv6 and you can see the point that (I think) GP is making.
Side note: The claim it is not widely used doesn't track. How many people use Google or Facebook? More than half of that traffic is over IPv6.
I know that the GP wasn't asking about static IP addresses, however I was merely providing some statistics about what customer asks regarding IP addresses are actually seen on the provider side of things.
It's not widely used in my customer base as well as among a number of other small independent ISPs I know of. A significant number of the CPE we have deployed today do not support IPv6, therefore a significant number of our end users will not be using IPv6 resulting in only partial usage. That is what I am referring to: an incomplete deployment of IPv6 across a customer base without their knowledge of change and the end user's ability to identify issues will cause support issues which costs money and time.
Because of how IPv6 is provisioned, going from a no IPv6-deployed state to fully-deployed state will encounter transition issues. It's not provisioned the same way as IPv4, and that's the whole bloody problem!!! Sure, I can turn on handing out IPv6 addresses from our edge routers via PPPoE today, but it's not going to "Just Work" for all customers without CPE config changes (assuming the CPE even supports IPv6 in the first place for which a bunch of them do not!). It's a complete pain that has less than zero benefit for a small provider today since everything is available over IPv4 and nothing is IPv6 only. Heck, another ISP I know had a customer ask for IPv6 support, and then the customer didn't even bother to provision it on their own router!
Another example: take Ubiquiti's AirCube product line. It doesn't support IPv6. To deploy IPv6 it would be necessary to reconfigure things to make the ONU for the customer act as the router for the customer's home network and place the AirCube in bridged mode. That's assuming that the ONU even supports IPv6 yet as it did not when I looked into it on the order of 6 years ago. And yes, there are a few dozen customers on AirCubes that fall under this case. This means moving into a territory where multiple different CPE deployment configs are now required for less than zero benefit.
That there are gaps in IPv6 support has been a problem, is a problem, and will continue to be a problem. The industry isn't 100% IPv6, and I don't see how that is going to be the case in the next 5 years or even the next 10 years. Consumer gear isn't there yet. That was the problem 5 years ago, that was the problem 10 years ago, that was the problem 15 years, and that was the problem 20 years ago... What's different today that's any different than yesterday?
It's easy to deploy IPv6 in data centers where you control all the infrastructure on servers where operating system IPv6 support was mature decades ago. It's not so easy as a small ISP when dealing with pile of random consumer gear that an ISP's customers bring to the table where IPv6 support remains hit and miss.
So when you said `ask for IPv6` you meant `ask for a static IPv6 prefix` or something else similar to a static IPv4 address? Or is this an apples to oranges comparison?
And then you say `Nobody asks for IPv4` - so nobody asks for IPv4 and 0.5% ask for IPv6?
2 of my customers explicitly asked for IPv6 prefixes. Another ISP had a couple of users actively using IPv6 15 years ago, but it bitrotted and nobody cared enough to ask for it to be fixed. Another user recently asked to be provided with IPv6 (actually, forcefully demanded it rater) and then didn't even bother to enable IPv6 on their router.
So, yeah, I don't see IPv6 being relevant as a small ISP today.
I have had a grand total of one customer ask in almost 10 years. I have the block and announce it but there are so many dangers with implementing it that I am scared to even try yet until we slow down a little.
Comcast was forced to go to IPv6 because they ran out of IPv4 addresses in the private address space to use for management of their network (think of how each and every cable modem needs a an address for management in addition to all the routers and CMTSes). I was a fly on the way inside one of the router vendors when this took place more than 15 years ago.
I'm not doing CGNAT. We were able to get enough IPv4 addresses directly from ARIN a few years ago after being on the waiting list for a couple of years. It's a pity that widespread fraud depleted that pool faster than it should have been.
CPE support for IPv6 has generally been garbage with it taking 15-20 years before the bare minimum was supported by mainstream router vendors. Even today there are still vendors that assume only IPv4 support. In my opinion the IETF really screwed up when they made IPv6 more complicated than just IPv4 with more address bits. The incumbent in my area generally uses PPPoE in their access network, but routers that supported PPPoE and prefix delegation basically didn't exist in 2010, and only started being available circa 2015 (in part due to the required bits not existing in OpenWRT and the hardware vendors' software development kits for their chipsets). Sure, we're 10 years further on now, but there remain a number of vendors that only support IPv4 for management of devices (cough Ubiquiti cough) in parts of their product line.
That said, there are features of IPv6 that are absolutely awesome for carriers. The next header feature that pretty much eliminates the need for MPLS in an IPv6 transport network is one such item that makes building transport networks so much cleaner when using IPv6 than IPv4. No more header insertion or rewriting, just update one field and fix up the delta on the checksum and CRC. They just aren't really applicable for smaller networks.
Ah, okay, well if you already have the IPv4 address space available for all your customers that's a different story. I can understand why you want to wait to dual-stack in that case.
I do think NAT64/464XLAT is a pretty good architecture for new ISPs that can't get their hands on IPv4 space, though. Or even MAP-T, but CPE support isn't really there yet.
CPE support is the issue and has always been the issue! People assume that IPv6 is done, but it really isn't in the consumer networking gear product space.
Android and iOS have pressure on app developers to support IPv6 or at the very least function on 464XLAT. On home broadband people could connect anything including programs that use literal IPv4 addresses and break on 464XLAT. Things like corporate VPNs, Skype, online gaming. Ironically these services would benefit the most from supporting IPv6 but they've evolved to deal with IPv4 NAT to the extent that they've become dependent on IPv4 connectivity.
For me, no IPv6 = no business. I don't think it's acceptable to build a network on IPv4 only at this point, it speaks to being willing to cut corners and not do things the right way just because it's easier.
I worked in this space for a while, in the US. Outside of the major cities, Internet service falls off extremely quickly. Like, shockingly so: you can be as close as fifty miles from, say Philly or Flagstaff and have zero fiber, zero cell coverage, just nothing.
The people who attempt to fill these gaps are commonly rural telephone companies, electric cooperatives, tribal entities, or mom and pop shops where the owner grew up on a Ditch Witch and only knows as much IP networking as essential to light up the fiber and get the packets flowing upstream.
They are enormously resource constrained in ways you might not expect, too, eg operations can grind to a halt because everyone is out with a chainsaw after a storm, or because the Guy that Knew Stuff about their network died suddenly.
They are very, very unlikely to decide to run an IPv6 network just because. There's no upside that makes the juice worth the squeeze for them.
In principle, IPv6 core networks can actually be very beneficial for small providers just starting out if they're not able to get IPv4 addresses for their customers and are forced to use CGNAT.
In an IPv4-only CGNAT setup, all the traffic has to flow through the CGNAT gateways, and that gear is stupidly expensive. Having IPv6 in the mix means that anything that supports IPv6 (such as most streaming services) won't hit the CGNAT gateway and can just be routed natively. This can really save money on CGNAT hardware.
For implementation, you can use NAT64/DNS64 for your CGNAT setup and implement 464XLAT on the CPE. This keeps your whole edge network IPv6-only so you don't have the complication of maintaining two parallel configurations on the edge.
There is also MAP-T, which is even lighter on infrastructure since it pushes all the state into the CPE and avoids the complication of stateful CGNAT. But unfortunately CPE support for it is pretty limited at the moment.
AFAIK in Germany for DOCSIS it's common to have DS-Lite in the mode with lightweight 4over6, so the CPE does the NAT and the carrier just range checks the client side IPv4 port to fall within that customer's block.
No state at the carrier; IP:6 native network.
If you buy a full IPv4 you get one for yourself but I expect still through the same tunneling machinery to keep the network IPv6-only.
> Outside of the major cities, Internet service falls off extremely quickly.
Saline is less than 10 miles from Ann Arbor.
> The people who attempt to fill these gaps are commonly rural telephone companies, electric cooperatives, tribal entities, or mom and pop shops...
That's fair, but at some point, you need to recognize you are competing with a major ISP. No one appreciates it when you come in, tear up the roads, and then pull out once the incumbent ISPs bump up their speeds ever so slightly. (Looking at you, Google.)
> They are very, very unlikely to decide to run an IPv6 network just because.
No one deploys IPv6 "just because," and yet more than half of the traffic to major sites is IPv6.
> Outside of the major cities, Internet service falls off extremely quickly
I live half an hour from a state capital and my only option is cable... the coaxial cable they laid bare on my flower bed decades ago. I dig it up about every other year when planting. It's not even in a conduit!
I wish I could say no IPv6 no business. There are only 2 ISPs here, one cable and one fiber. Neither have IPv6, the smaller ISP also does CGNAT because IPs are expensive.
I'm trying to convince them that they could save money with less powerful CGNAT hardware if they deploy dual stack.
I agree in principal but if the only other option is Charter/Spectrum/Comcast, you bet I'm going with the "lazy" person's fiber.
I have spent most of my career under the thumb of fucking cable and I'd sooner slam a car door on my nuts than go back to paying so much money for such garbage service.
Everybody can muddle along without IPv6, so it's easy to make it a very low priority. Especially for small shops that are struggling just to create a viable business. IPv6 needs something more to motivate it, a web destination or application that is only available on IPv6.
> IPv6 needs something more to motivate it, a web destination or application that is only available on IPv6.
How about not having to pay for (as) beefy CG-NAT hardware because people that go to Youtube, Netflix, MetaFace, TikTok, etc, can directly connect via IPv6.
Hadn't thought of that, but it might not be a huge savings unless you were to go ipv6 only. If you're still going to support ipv4 anyway, the hardware savings might not be too significant.
> Hadn't thought of that, but it might not be a huge savings unless you were to go ipv6 only.
Even a small number of devices/services not supporting IPv6 can have huge costs:
> Our [American Indian] tribal network started out IPv6, but soon learned we had to somehow support IPv4 only traffic. It took almost 11 months in order to get a small amount of IPv4 addresses allocated for this use. In fact there were only enough addresses to cover maybe 1% of population. So we were forced to create a very expensive proxy/translation server in order to support this traffic.
> We learned a very expensive lesson. 71% of the IPv4 traffic we were supporting was from ROKU devices. 9% coming from DishNetwork & DirectTV satellite tuners, 11% from HomeSecurity cameras and systems, and remaining 9% we replaced extremely outdated Point of Sale(POS) equipment. So we cut ROKU some slack three years ago by spending a little over $300k just to support their devices.
Sadly the post is now behind a login: what happened later was Apple donate a bunch of Apple TV devices to the tribal ISP and that cut their IPv4 usage by an order of magnitude (or some ridiculous number) and there were major savings. The ISP then recommended AppleTV to all of their customers to get the best viewing experience (because of the latency/overhead of CG-NAT when streaming video).
So the more you move over the more headroom you have for the broken IPv4-only systems. AIUI, the rollout of MAP-T/E has helped in that things are more stateless, and more work is done at the CPE, but there's still overhead.
We used to have freeipv6porn.com, lol. But I suspect that was a joke as much as anything else given how much porn you can get for free all over the Internet.
Thankfully, they are doing IPv6, although one day I had some weird issue where IPv6 was broken but if I disabled it ipv4 was still working. Could have been my fault, IPv6 is generally new to me (not much of a network person).
I get the impression that they are still learning to run an ISP, both technically and customer facingly. It's weird - I learned more about them from this article than from actually being living here with them.
Surprised they aren't deploying NAT64/DNS64 with 464XLAT on the CPE. You get essentially the same setup as CGNAT for IPv4 services but your whole core network is native IPv6 so you only have one set of address space to manage and your customers will be able to directly connect to anything IPv6 related.
Charter does too except their allocation size is your puzzle to figure out. Maybe it's /56, maybe it's /60, maybe neither of those will work. I've run into all the above.
> Checking Comcast's site now, it seems that they now offer "unlimited" data. Interesting, that option wasn't there 6 months ago.
It's been there since they announced the data cap. I thought the unlimited bundled with leasing their higher end hardware came first, but the email from 2016 announcing that our plan was getting the cap mentions being able to pay for unlimited.
You've always had the _option_ of paying extra for unlimited data, however its only in the past month or two that they've started offering unlimited data as standard (in select markets).
I’m sorry I still don’t get it. Could you explain that in different phrasing ?
A comcast customer always had the option to pay for unlimited data. I get that part. What is the 2nd part? “Started offering it as standard” means what?
In markets where Comcast has actual real competition, they "include" the unlimited data (aka no cap) with no extra charge when you sign up for their gigabit plans.
You pay for their highest tiered and highest bandwidth plan and they have the audacity to impose a cap making that bandwidth work against you? Crazy. My household internet usage is quite modest, nothing anyone on HN would call data intensive—casual video streaming being the lion's share, and I blow out 1TB every month without fail.
In the Bay Area Comcast offered (2017-2023 at least) internet with a default 1-1.2TiB/mo data cap that you can lift for the month for an extra 10-20usd (I don't recall, my roommate who played CoD was the one paying for this by himself on every month with huge updates).
There's barely any competition here. You can pretty much chose from Comcast Business or XFinity, which both are just Comcast because of a free market with free as in not in jail.
It used to be 30/month for me. I was not renting their modem and got charged more for it.
If starlink ever gets more capacity, I'll probably switch. Right now I think the only way to get gigabit down on starlink is with four or five accounts and manually bonding the dishes together. As soon as that obstacle goes away, Comcast will have competition in my area and I intend take advantage of that.
What ISPs offer and how much they offer it for tends to vary wildly region to region.
If you live in a region where they have no meaningful competition (which is still fairly common in a lot of places in the US) well bend over and lube up.
They'll vary wildly, as much as they think they can get away with, in the hopes that you will never use the service and pay them as much as possible for it, and they'll bury your mailbox with crap to try to wear you down into coming back.
They will happily let you pay for years, for services that no longer exist, no longer connected to any of their networks. They'll take you to court rather than pay anything back; they know they are receiving extra money, and there's a significant amount that comes in, but "oh, it's so confusing, and there are so many legacy systems, we can't possibly catch every mistake."
The money they shuffle back and forth between each other, daily, reeks of book cooking - you might have a stretch of 20 miles of trunk in which there are 20 separate owners - not concurrent riding separate fiber lines, but in sequence, each paying rent to or getting rent paid by the adjacent rider, even though only a single company actually services the entire span.
It's funny how construction companies and ISPs get these rackets going, and then when people come along like these PrimeOne guys and offer a reasonable rate on a decent product, it's somehow vastly disruptive and threatening.
They'll expand, and be encouraged and allowed to expand, and after 5 or 6 years, the big ISPs will start circling, and eventually buy them out, and they'll retire happy. AT&T or Lumen will own their network inside of 10 years, and they'll claim it's modernized and upgraded infrastructure. People with shitty oversold undermaintained cable internet will be left alone until the money stops.
Starlink to phones is great, if it only didn't make ISPs so much money handling the base stations on the ground.
There's fiber all over the US just hanging there, unused, unmaintained, because merger after merger after merger left giant piles of assets under the ownership of companies like comcast and centurylink and at&t, who left infrastructure to rot, often built with public funding, and maintained their local monopolies and shitty service.
Whatever it is we're doing to regulate the industry at a federal level isn't working, but I imagine that's where a lot of the money goes.
A healthcare company I used to work for had a location in CA that, on paper, had 3 circuits - AT&T, Verizon, and Comcast. Each had their own Ciena box in their telecom closet. They were getting a circuit upgraded so one of them was newly installed and we needed to cut over the slowest one to an upgraded one.
Working with the director there, IIRC we traced down the Verizon and Comcast box as actually being connected to the AT&T box. After some checking of circuit IDs on boxes it seemed that the LEC for both the Verizon and Comcast circuits was AT&T, and AT&T was the actual owner of that single physical fiber going out there.
I wish I could sell the same thing 3 times, lol. Contracts were signed on all these circuits so there was no getting out of it.
Yeah they’ve offered unlimited for a few years now for an additional charge. Now as of last month all plans are unlimited if you just update to them. It wound up being cheaper for me too.
However I still applaud these guys. There needs to be more competition.
Comcast similarly removed their 1.2TB cap in my neighborhood within months of us getting fiber. It's almost like the only reason for the cap was because they could get away with it when there wasn't any competition.
Comcast is notorious for exploiting places that don't have any other real options. Just before Google Fiber was activated in my area, Comcast stepped up their game big time. The only problem is that they had spent years nickel and diming me for actual connection speeds that didn't even come close to their advertised rates, and their latency/jitter is garbage compared to fiber. Comcast clearly doesn't want to have to compete. In their defense, their connection was rarely down.
When I lived in downtown Oakland CA, Comcast literally could not keep up price-wise with the competition. Their customer service jaw would drop when I told them our local fiber offered a flat fee cheaper than theirs for 10 gigabit symmetrical fiber. On top of that there was another local microwave wireless option that wasn't too terrible.
The only thing in the end their salespeople could do was offer TV bundles but still wasn't cost-competitive. Not sure what their offerings are now but it was such an easy decision to switch.
> is notorious for exploiting places that don't have any other real options.
Isn’t this standard competitive practice ? Charge what the market will bear.
I don’t know if I’d call that “exploitation”. If there’s one gas station 90 miles from every other gas station in the Nevada desert, they’re gonna charge more, aren’t they?
Yes, it certainly is. But isn't it interesting that Comcast is almost universally hated? I used the word "exploit" simply because had they treated their customers better and focused on putting their best foot forward, I don't think they would have bled customers nearly as quickly.
I live on a dead end road with two ISPs and the one that isn't comcast is unusable. Comcast pretty much laughed at me when I asked if there was anything that could be done to drop my overpriced bill
Correct. It was a very calculated decision. They were squeezing out profits by trying to move heavy users to the next tier of service. But this only works if they have a monopoly.
Feature-wise it doesn't matter because you're still going to have to play the price haggling game. Other providers don't renegotiate every 6 months like they do. They have more in common with Waste Management than with a respectable ISP.
I'm on the other side of the country and was a Cox customer for over a decade until they decided to add a data cap to their plans. Fortunately, wyyred rolled into town right around the same time, offering fiber at higher speeds, no data caps, and half the cost. It was an easy decision. I also noticed that Cox is now advertising unlimited data for free. Too little too late.
I love how these guys are trying to sort the lack of competition in broadband, stood up and did something about it and all the geeks on HN are upset that they are not doing something that the majority of their customers would not give shit about or even come close to understanding.
Everyone here that has started a company to challenge the entrenched monopoly raise your hands please.
I understand the tech deeply and that does not translate to the practical needs of trying to run a successful business.
Given they have a hundred ish customers its likely at most a couple would even know how to do that, or care. It's just not worth their time to worry about this right now. They seem to be doing just fine
My fiber installer referred to the Adtran 632V ONT he installed as the "modem".
He installed two other junction boxes (one outside the house near/under where the fiber attaches to the wall of the house, one inside near the ONT) but they're just passive optical couplers allowing them to swap out fiber segments in the event of fiber damage without re-running the entire install.
Can’t speak to this exact circumstance, but more generally: The ONT translates the SFP+ networking to fibre optic, but the modem is still somewhat necessary for logins if you use PPPoE as a wrapper for example. In telecom fibre optic, it often also assigns a particular vlan to internet packets and separate vlans for TV and phone. But I’m not an expert here, just explaining why I needed a modem function in my router as well as a media converter to house the ONT.
As far as I know, nobody uses separate boxes for the modem and router, that kind of thinking died when wifi became more widespread and included by default with ISP plans.
I wouldn't really call that a "modem" though, it's not really doing modulation/demodulation work to convert between media types. The terminology I usually hear for the provider's box handling any final authentication and VLAN splitting is usually a "residential gateway", which can be configured to bridge to a client's equipment.
Definitely splitting hairs here though on terminology.
The component that does the PPPoE and VoIP VLAN is typically just referred to as a router (or a "residential gateway" for companies that want to sound less technical), I've never heard as it referred to as a modem, usually the ONT is referred to as a modem as it's MODulating/DEModulating the optical signal.
This indicates that their local and state governments aren't (at this time) captured by the incumbent cable provider.
A captured state gov will pass laws to thwart new infra deployment, commonly written by ISP interests. A captured local gov will never approve deployment or slow-walk permitting in an attempt to bankrupt the upstart.
more explainers: New suburban fiber infrastructure means either trenching or pole hanging. The local gov issues permits for both but poles also require the cooperation of the pole owners. This last adds the PSC to the mix.
Recalcitrant pole owners are known to stall and kill infrastructure deployment - especially where going underground isn't an option. Some PSCs mandate that pole owners cooperate. Some PSCs abdicate that responsibility and are examples of regulatory capture.
I’ve been hearing about “captured government” with respect to fiber deployment for two decades now and the folks on that soap box have made absolutely zero progress on improving deployment of fiber infrastructure in that time. Tilting at that windmill isn’t working, because for the most part that’s not the real problem.
Why isn’t the Bay Area a hot bed of fiber deployment? You think Comcast in Philly has more pull with Cupertino and Mountain View than Google and Apple? No! Internet in the Bay Area is shit for the same reason all the infrastructure in the Bay Area is shit. The government makes it slow and difficult to build anything.
Comcast installed fiber to my house back in 2018 or so. The permitting took months. And this was to run Comcast fiber on poles where Comcast already had their own cable lines. And my county is actually pretty efficient with permitting. It’s just that American municipalities absolutely hate it when anyone builds anything.
I guess Comcast doesn't need to capture the local government in places where it's already illegal to build anything. But in other places it has definitely happened.
In most places, permitting is the most direct and immediate roadblock. If you don't start there, you won't make any progress on the issue.
I live in a blue state that actively encourages municipal and cooperative fiber deployment: https://mdbc.us. It's had approximately zero impact outside some rural parts of the state.
If you think internet in the bay area is shit then you haven't seen how bad it can get. Even other large cities within California like LA and SD are worse.
Having lived in all three mentioned areas, none seemed particularly bad unless your standard is fiber straight to home. And seemed like Bay Area had more of that if anything.
I am absolutely not defending Comcast here but its worth pointing out that every anecdotal bit of evidence about performance and reliability can be true.
Network performance in that last mile can differ by block and even by season. An otherwise functional run of coax might have intermittent ingress but only shortly after it's rained while cold out.
This isn't even counting all of the flaky performance anecdotes that really boil down to overcrowded Wi-Fi or poorly configured consumer gear or anything else that isn't strictly the fault or problem of Comcast.
In college, we had the combination of spotty wifi + 10 dudes concurrently torrenting Shrek 2 + jank Android tablets running rogue DHCP servers and taking router's IP + rats chewing through cables, yet every time the internet went down, they called Comcast. And it was never actually Comcast's fault.
Maybe it's good enough that not very many people care. I moved around San Jose, Mountain View, Berkeley, and Sunnyvale, never noticed problems with Comcast or AT&T. Was expecting flakiness after hearing all these bad stories, but no, it was reliable.
What you don't get often is fiber-to-home, or great upload speeds. But most people aren't running big home servers.
CGNAT isn't a matter of cheapness. There are more Internet users than IP addresses. Someone has to draw the short straw, and that's the providers that came late to the party.
Now, if they're doing CGNAT and also don't have IPv6, that's just crap. They shouldn't be allowed to call themselves an ISP at that point.
This was even a major hurdle for Google Fiber. The incumbent ISPs did everything they could to obstruct them from installing fiber, and it was fairly effective even against someone with deep pockets like Google.
Looks like they're somewhat rural which probably makes it way easier. I was a project manager for a Telco years ago and the process to get fiber run in an established city is crazy. Had no idea how much was going on under the roads until I had to plan out conduit boring projects.
It's not that easy. Poles vs trenches are a tradeoff discussion. FWIW I was once in construction digging trenches and I'm German, so I might be biased a bit.
Pro poles / open air:
- very, VERY cheap and fast to build out with GPON. That's how you got 1/1 GBit fiber in some piss poor village in the rural ditches of Romania.
- easy to get access when you need to do maintenance
Con poles / open air:
- it looks fucking ugly. Many a nice photo from Romania got some sort of half assed fiber cable on it.
- it's easy for drunk drivers, vandals (for the Americans: idiots shooting birds that rest on aboveground lines [1][2]), sabotage agents or moronic cable thieves to access and damage infrastructure
Pro trench digging:
- it's incredibly resilient. To take out electricity and power, you need a natural disaster at the scale of the infamous Ahrtal floods that ripped through bridges carrying cables and outright submerged and thus ruined district distribution networking rooms, but even the heaviest hailstorm doesn't give a fuck about cable that's buried. Drunk drivers are no concern, and so are cable thieves or terrorists.
- it looks way better, especially when local governments go and re-surface the roads afterwards
Cons trench digging:
- it's expensive, machinery and qualified staff are rare
- you usually need lots more bureaucracy with permits, traffic planning or what not else that's needed to dig a trench
- when something does happen below ground, it can be ... challenging to access the fault.
- in urban or even moderately settled areas, space below ground can be absurdly congested with existing infrastructure that necessitates a lot of manual excavation instead of machinery. Gas, water, sewers, long decommissioned pipe postal service lines, subways, low voltage power, high voltage power, other fiber providers, cable TV...
There's a huge downside to poles where I'm based: permit shenanigans by pole owners that delay projects and allow incumbents to destroy competitors. Granted, some municipalities do the same thing. One local municipality I have to deal with responds to permit requests almost instantly, while another takes weeks of pestering to acknowledge even the most basic of permit requests.
For anyone starting out today, I would strongly recommend having a planned legal / regulatory strategy to fall back on in the event that excessive delays occur by parties you cannot avoid dealing with.
Bell techs have done that and caused outages for customers a number of times. One outage ended up costing close to CAD$7900 to repair that they settled for due to it occurring on a weekend in a rural area during a snowstorm that required third party traffic control to meet the Book 7 requirements (on a weekend at the bottom of a hill in a snowstorm with insufficient visibility for us to work safely while doing a road crossing). Normally that would only be a $1k-2k or so, but traffic control, lack of slack due to the location of the cut and overtime for all parties involved really drove costs up. Less than 30 seconds with a pair of wire cutters can be downright brutal to repair if the conditions hit the worst case.
Meh, here in Germany you got the same issue with trenches. It takes ages to coordinate digging them, I think the worst example simmered for two years until the permits arrived. And then, it's a nightmare because you can't just cut off people's courtyards and parking spots for any time longer than absolutely required, so as soon as you're at depth you gotta cover the trench with steel plates so cars and pedestrians can cross...
those days it's not trench digging (unless it's next to highway with machine that in one pass will trench and lay conduit/cable), it's trench drilling with something like this https://www.ditchwitch.com/directional-drills/
frontier installed fiber in my area using this method. relatively quick and no damage that needs to be "aggressively" paved over.
Can't do that in a century old city street with century old utilities in it.
They dug up the sidewalk with shovels here as a result; though using machinery to move the soil they dug up and handle the asphalt that's covering the sidewalk along part of the street.
Yeah you can directionally drill across a front yard typically; but actual urban areas are just too filled with infrastructure.
before drilling there was a team that mapped and marked all underground utilities.
some of the drilling went between front yards (this is where underground vaults are/were), but a bunch was done below sidewalks and across the streets.
I am in a rural area of Texas and I just recently got access to fiber. The other competition is ADSL and DOCSIS providers - AT&T and Optimum.
Optimum had their entire service area bought out by Comcast the day after I switched. Comcast has since broken every major utility at least twice and my fiber connection three times by working on the old infrastructure. I think Optimum won that trade. I can't imagine many residents are going to prefer Comcast over $80/m for no-bullshit internet, especially after the water main break they caused last week.
These FTTP providers have the game solved in Texas. I've seen them do 500-1000 homes in <30 days. Their directional drilling expertise and aggressive neglect for 811 seem to get things done very quickly. There are some areas with competing fiber providers now. I've got 5gbps symmetric for $110/m and I live in the woods. Trees go through power lines and the fiber infra is completely unaffected. The only utility left to bury is the electricity, and they're actively working on that in some areas now.
Texas regulations are quite something. My friend told me that the closest thing to regulation and zoning they have is... an HOA. This was the first time in my life I heard anything positive about HOAs.
Still waiting for someone to do the same in Bay Area. Many parts of it don’t have any fiber optics options, even though Sonic does provide some in the north.
AT&T put an optic cable at my curb 10 years ago (most likely due to imminent competition from Google Fiber internet), but then never lit it (most likely because Google dropped their effort due to complications with cities)…
Sonic is doing this in sfba. Used to be att reseller now they lay their own fiber, 50% cheaper plans, byo router, ipv6 that actually works, great service.
Online gaming: No "strict NAT" program. No conflicts with having multiple consoles at one locations. No port forwarding needed.
VPNs: Business IT usually uses some weird IP range like 172.xxx.xxx.xxx so avoid conflicting with the popular 192.168.1.xxx or 10.xx.yy.zz. When two companies merge there is now an IP range overlap and a renumbering has to be done.
P2P file transfers: No port forwarding needed.
Self-hosted servers: No port forwarding needed.
Video chat/VoIP: One reason video chat still suffers from bad quality is that video is proxied through cloud servers to deal with NAT. With more IPv6 video chat services can use more direct connectivity lowering costs and improving quality.
Avoiding CGNAT. If anyone on the same CGNAT group in your ISP got banned you will also be banned. Many Internet services don't have IPv6 and they often cite IPv4 based reputation as the reason why they won't deploy IPv6.
> When two companies merge there is now an IP range overlap and a renumbering has to be done.
That has nothing to do with OP's question about consumer internet concerns. However, I've been through several corporate acquisitions. The thing that took the longest to work out? Getting the new company's name on the paychecks.
The thing that took the second longest to work out? Renumbering networks and consolidating and decommissioning hardware for which there just wasn't enough RFC 1918 space available. Every. Damn. Time. this process dragged on and on and on, when IT could have saved 6->18+ months of future work [0] by generating a /48 ULA and using that for every internal thing but the stuff that was IPv4-only.
[0] ...and god only knows how many dollars in labor costs...
The big customer-visible feature is that every machine on their LAN can be globally reachable. For ISPs that absolutely cannot get enough IPv4 space to give their customers even a single globally-reachable IPv4 address, this would be the only way to get any global IP space. IME, edge routers that are intended for networking noobs to use often set up their firewalls so that they block inbound unsolicited IPv6 traffic, but (unlike with IPv4 NAT) there's the option of opening up multiple hosts to some or all inbound traffic.
Another feature that I find to be pretty stupid (but that some folks seem to really like) are IPv6 "privacy" addresses. Because each host usually is assigned an IPv6 address in a subnet that's 64 bits wide, most mainstream OS's have configured their IPv6 address autoconfigurator to set one stable, "permanent" address, and to set a parade of periodically changing "temporary" addresses. The OS is usually configured to prefer the permanent address when software asks for a socket to listen on (and sockets that handle replies to that listen socket), and those temporary addresses are preferred for sockets that initiate outbound traffic. The idea is that this is supposed to confuse tracking, but I'm very skeptical of its efficacy in the real world.
Finally, a customer can also usually get enough IP space to make globally-reachable subnets on their LAN. Depending how the ISP has configured things, a customer can get between four and 256 subnets. These subnets are handy to provide networks that provide globally-reachable IP addresses, but that can be easily logically isolated from the rest of the LAN by the router.
My feeling with privacy extensions is that it's more to prevent enumeration of a network by looking at outgoing traffic. It may have a degree of tracking protection too, but that's probably less important, and is about equivalent to everything being behind a single NAT address.
The RFC 4941 problem statement [0] notes that an attacker who can look at outgoing traffic from a network will not be thwarted by "privacy" addresses. The background section of that document goes on to say that "privacy" addresses do little-to-nothing to thwart correlation attempts against client networks that have few hosts on them.
IMO, something like what's described in RFC 7217 [1] (changing the interface identifier used for "permanent" addresses from the interface's MAC address to something that mixes in the advertised prefix) is a much better way to address the concerns described in section 2.3 of RFC 4941.
For regular folks there isn't much benefit tbh. Mainly I think it simplifies ISP architecture and offers slightly faster (like 10%) performance but ISPs have to support IPv4 stack for foreseeable future anyway so kinda moot point. If you game a lot p2p (i don't) you should, in theory, see lower lag.
For me personally, I work on networking startup so I'd like to be able to run IPv6 stack from my home network to test things.
If you don't currently have IPv6 service, have you looked into something like Hurricane Electric's IPv6 tunnel broker? [0] It's how I got my first IPv6 subnet, and worked really well for me. I stopped using it when Comcast finally got around to providing IPv6 for non-business accounts.
It has also been about that long since I used the service. According to my very, very distant memory this requirement was implemented by having you punch in your IPv4 address into a form to configure their end of the tunnel. I know for an absolute fact that Comcast was not providing me with a static IPv4 address, and the HE.net tunnel worked just fine.
Unfortunately Sonic does not cover the whole bay, and certainly not all SF. I am still waiting for Sonic to cover the heart of the City (Eureka Valley).
Downtown San Jose is nice - I have fibers from both AT&T and Sonic. I switched from AT&T to Sonic a couple years ago and have been impressed. I pay half what I did, get 10x the speed, and customer service is much better.
In my condo complex the cost for Sail is the same as the lowest Comcast internet plan. Possibly Sail and others have forced Comcast to go down finally, which is great.
Also the Sail support is really good, fast responses (even the CEO sometimes answers tickets).
I've seen a few articles about folks who started an ISP and they always talk about the physical infrastructure. But in today's world where ISP ads are touting the speeds of their wifi, it really makes me wonder what the support burden ends up being like. What's the breakdown for actual ISP issues vs issues with customer equipment?
My experience from almost a decade ago, mostly in DSL land, is that most customer calls were "my WiFi doesn't reach through the solid steel wall the router is hung against" and "how do I set up my email" and maybe "I lost the password to my WiFi again". WiFi issues were especially bad when 802.11n got finalised but there were tons of "draft n" WiFi devices out there that almost followed the WiFi spec. I still shudder when I see Atheros listed in device manager.
There were things that made the ISP I worked at special, one of them being that we pretty much defaulted to having customers hook up their own DSL, which meant spending a lot of call time helping people who have no idea what an RJ11 jack is install plugs and adapters.
I've also spent a lot of time on "the password I use for my email doesn't work on my Facebook" and "my USB printer doesn't work". People don't know who to call for tech support so they try their ISP. There was also the occasional "the internet is broken" whenever the user's home page had a different theme or design as well, those usually came in waves.
Once the modem and/or router is installed, most internet services Just Work. There are outages and bad modems and the occasional bad software update to deal with, but they're a relatively low call volume compared to what customers call about.
I quite enjoyed "I run a massive home business and require 24/7 uptime and will get extremely aggressive about this"
"Maam, if your business is that important, surely you as a responsible business owner have gone and purchased a business class internet service with 24/7 SLA. It says here, you are on our cheapest, residential VDSL service"
Which is why comcast goes to such great lengths to ensure they own as much of your network stack as they can - in my area at least, their support is capable of fully managing your router and WiFi remotely if you're leasing their equipment. I imagine this is a great boon for their ability to provide tech support (and includes a host of other "features" that don't serve direct customer needs such as a non-optional guest WiFi access point that any other comcast user can use).
This leads to fun tech support calls if you use your own equipment where you're basically proving to the support underling that you know how to run your equipment for the first 20-30 minutes before they take your issue seriously (yes, the modem light is green, yes, I've already power-cycled, yes, I'm testing on a wired connection, etc)
> Great. Glad to hear you are connected via hard wire Mr. teeray.
> Please wait a moment while I check on some things on your account.
> Thank you for your patience. Can you please confirm for me that you see a green light on the top of the device? Can you tell me whether the light is blinking or is solid?
I know for a while (I switched back to consumer a few years ago) Comcast Business let you persistently opt out, but if you opted out, you couldn't use other people's APs (either "share and get access to that network" or "don't share, and don't").
Back when Comcast made it absolutely mandatory to have a technician come to the house to do the install, I just chatted with the tech about computer networking and our respective home setups. This usually got me the phone number for the local tech support office along with a "Call this if the service is giving you any real issues.".
> This leads to fun tech support calls if you use your own equipment where you're basically proving to the support underling that you know how to run your equipment for the first 20-30 minutes
For analyzing support burden, I think the relevant question here is why have you even had the experience of calling tech support for a non-working connection - and that falls squarely on the non-reliability of Comcast's network.
Exactly. That's the kind of logic that only makes sense in a metastasized corpo. The only times my non-incumbent fiber connection has gone down in 8 years have been overnight maintenance windows that only happen maybe a few times per year.
This is 10 years out, but I used to work on an IT help desk, that was the outsourced 24/7 helpdesk / hosting for a collection of small local/regional isps (<5000 customer rural dsl companies, local municipalities, apartments, etc) My ballpark estimate from that over 3 years working there is probably 75%+ are Not equipment related. Setting up email was a big one, people accidentally hitting the input/source button on their remote and losing their STB input setting, People needing to reboot their router, flushing DNS settings / winsock reset. These might have been the majority of cases.
other than flushing DNS / winsock resets, I don't understand how the rest of those are blockers.
I think my conception of basic tech illiteracy among the general public is vastly wrong. I generally like to believe most people are competent enough to handle these sorts of things.
Those aren't mutually exclusive things. Even if 99.9% of Comcast customers are pretty good with technology, and only 1 in 1000 customers are illiterate enough that they have trouble selecting the correct input on their TV... with 32 million customers, that means you might get tens of thousands of calls about it.
But really, internet (and digital TV) services are pervasive enough that they are no longer just for technologically inclined and resourceful people. All aspects of society are now using the internet, even the homeless, impoverished, disabled, and institutionalized.
I once spent half an hour on the phone with a dialup customer who couldn't get connected. Fortunately, they fixed their own problem - they had been entering the "letter zero" instead of the "number zero" (their words.)
Took another call from an irate dialup customer who demanded a refund - he didn't know he needed a computer to use the internet -- and had driven himself mad dialing up our modem bank with his telephone and waiting for the training tones to subside so he could begin to navigate the internet as he imagined it to work: press 1 for email, 2 for news, 3 for weather...
Despite the proliferation of smart phones and greater prevalence of home networks, I don't think the situation has changed much for a segment of the population once you get down to troubleshooting why something isn't working. The skills and the willingness to just try to fix the problem aren't there.
ISPs are weird: You don’t call the water department if your sink is backed up—you call a plumber. You also don’t call the electric company when you want to wire your finished basement—you hire an electrician. ISPs somehow became responsible for absolutely every aspect of consuming their service though. Why isn’t “home internet plumber” a thing?
Most people don't have the equivalent of home internet plumbing in general. They have a hole drilled into the wall (by the ISP) where the all-in-one modem-router-switch-wap sits on a shelf. There's probably a third party service to get ethernet run through your walls, and maybe even replace your all-in-one box with something good, but most people are just doing the equivalent of getting water straight out of the water company's tap with no plumbing.
This, and also, it's much more common for internet problems to be caused by upstream issues not in the house (partly because of the situation you describe....not much to go wrong on the users end). It's very rare that a plumbing problem is because the main water line lost pressure.
Back when I still had ISPs that provided the modem + router, every single issue I think I ever had fell into one of two categories: a modem and/or router power cycle fixed it, or it was a broader network issue that had nothing to do with me or my particular internet situation (this is omitting the most common third issue: terrible customer service problems, but that's a separate thing)
After fixing internet for some neighbors and older relatives, I've wondered if people would pay for a home network / internet handyman service. It's super frustrating, especially for older folks. They often confuse their email passwords, ISP passwords, wifi setup, etc. Also I could save them a bunch of money getting rid of services they don't use, like moving their landlines to VOIP.
VoIP doesn't necessarily require your router to be up.
For example, if you subscribe to Verizon FiOS voice, the technician will disconnect your copper phone lines and connect them to VoIP termination on your ONT.
This is true, but it's not just that. How many useless cable TV packages are people paying for, on top of Netflix, Hulu, and tons of other streaming services?
I had the same thought, and even took on a few "customers" (local folks I didn't charge, but used as a test group). If I decide to do it "for real" I will definitely need to build a relationship with a person who can run ethernet cables through walls for people. I can do that, but the time it would take would not make it worth it for me.
I don’t understand what you mean. If you want Ethernet run through your house, or coax in more places, or access points mounted, you don’t call your ISP.
You call an electrician or a handyman or somebody and tell them you have some low voltage work.
The ISP provides a cable box and modem to most homes in the same way that the electric company sticks a meter on your wall.
Having worked with the public before, I have no doubt that a lot of people likely do contact utility companies for issues inside their home. Some of them even do have repair programs with outside contractors. People often simply call whoever they have an existing business relationship with for issues related to that product/service. It may be ignorant but it isn't illogical.
Also, as the other commenter pointed out, ISPs don't terminate their service at the edge of your premises. Basically all of them today will connect one of your devices to confirm installation.
There are times when you're better off calling the local sewer department first.
In San Jose, if you see evidence that your house's main drain has backed up and you have a cleanout within 5' of the sidewalk, you're better off calling the city first before calling a plumber -- the sewer department will snake the "lateral" pipe between the cleanout and the main sewer line under the street for free.
The one time we used this the response time was very quick (in line with the 30 minute response time they cite on their website).
Fibre is orders of magnitude better than DSL or cable as entire classes of problems are eliminated. Water shorting out copper pairs? Not a problem unless the water gets inside a splice and freezes causing significant bends that lower signal levels. Water getting into a cable is generally not an issue as most cables are either gell filled or have water blocking tapes. Lightning strikes are generally a non-issue since the cable isn't going to conduct a damaging charge into the ONU/ONT.
With careful selection of the customer ONU/ONT, the incidence of support calls means that it can be weeks between customer issues on smaller networks. These days my biggest support headache is in house wireless coverage. It's also the one part of internet service that most people are unwilling to invest even small amounts of money to improve. The worst are the folks that install outdoor wireless security cameras without thinking ahead to putting them on a dedicated network to avoid driving up airtime usage and congesting the main wireless AP.
Right now device manufacturers are capturing the sliver of the market that actually cares about having fast Wi-Fi. A few ISPs have partnered with these manufacturers through the time honored business model of charging $10-20/mo forever on a box that costs $200.
My electricity and water is much more reliable than my Internet service. Then again, I've never called my ISP about an issue that wasn't 100% on them, but the HN crowd is more exceptional in that sense than most people.
In my experience with small ISP's they make 99% of their own headaches, the last 1% being provided by the small consultants who work with small ISP.
For fiber customer side issues are almost all wifi related, to the point that some operators will offer in home managed wireless options.
I used to provide support in an area where a provider had purchased a VDSL network in order to convert those customers to fiber. 20% of customers remained on VDSL for various reasons. 10% of customers had been moved to a dodgy hybrid fibre/last mile ethernet solution. and the remainder were all on fibre.
70% of support issues related to the VDSL customers. 20% the ethernet customers. and the remainder of issues were almost all wifi or power related.
They had a policy of charging customers 1000 bucks or so to convert them over to Fibre. Eventually they sold the business to a larger entity. 4 weeks of VDSL complaints, and the new owners gave everyone remaining on copper a free fibre upgrade.
Actually it was only technically VDSL. What they did was drop a fibre ntd into the old vdsl node, commission each port for a different customer, and then run a Ethernet / VDSL converter over the old lead in. The "upgrade" was just using the copper as a draw wire for the fibre cable. Nothing over 100 meters.
I'd imagine it's a lot less than "Okay, let's start by going into your dialer settings..."
With fiber, the ISP can see that everything is good up to the GPON terminal. Probably the router too as most customers will just use the ISP provided one. So that leaves the ethernet interface / wifi card as the only thing that would fail and have to be ascertained over the phone, and with a local ISP its probably more cost effective to cut out all the abstractions and just have a tech stop by to check it out.
On the other side, customers have become a lot more used to self help. For example their email isn't even hosted with the ISP any more! I would think that most people would be aware that if a device works good close to the router, and not good far, the issue is wifi range. If they're still calling the ISP, you can direct them towards wifi extenders. Or if device A does not work but device B does, it's not a problem to call the ISP about. And so on.
Of course this is my idyllic view not having worked ISP tech support in a few decades...
I am always baffled by these things. Say there’s a huge company with a monopoly in your area. My first thought is “How did they get that monopoly? What happened to all the other people who must surely have had the idea to compete with them?” But no, these stories are always treating “Hey, let’s start a competing company!” like some revolutionary idea that nobody has thought of before, and that success is assured.
There's a huge gap between "had the idea" and "had all the technical skills, the $millions in capital, and the managerial ability to actually build it". Then there's the barrier of "and succeed". If you read between the article's lines a bit - these guys had loads of the first 3, yet they're still losing loads of money every month.
But, bigger picture, you have a good point. These articles are obviously cherry-picked stories, with an extremely optimistic "... and the little guy wins!" spin. Ars is writing for an audience of techies who are frustrated with crappy ISP's.
The capital for an ISP is surprisingly low. The main problem is getting a physical connection to your customer's house. And that's such an obvious legal minefield that no networking nerd wants to do it.
The capital can be low. Vs. the article notes that these two guys have 75 miles of fiber installed (to 1,500 potential-customer homes) and 15 local employees. Vs. currently monthly revenues of about $10K.
Yeah, obviously these guy's long prior experience - pulling fiber for other ISP's - was another critical cornerstone of their ability to go from idea to build-out.
In the specific case of ISPs I think it's always because you won't make enough money to justify it as a big company, yet the task is too big and complicated to do it as an individual nerd.
The worst part appears to be the physical wiring. If your government has implemented loop unbundling, you're already set (probably need to do some bureaucracy and pay some affordable-at-a-stretch fees to get access to it). Otherwise, or if the loops are just crap, you have to figure out how to physically get a cable to everywhere, a task that is fundamentally laborious and legally fraught, not nerdy at all (unless lawyers are nerds) so nobody wants to do it.
Wireless ISPs are about as popular because of this. Wireless service is always worse, but you only have to install plant (physical infrastructure) at the customer's house and one central location, not all the places leading up to the customer's house. This makes it a whole lot more amenable to individual-nerd or handful-of-nerds setup.
I encourage everyone to at least think about how they would do it.
> ...you only have to install plant (physical infrastructure) at the customer's house and one central location, not all the places leading up to the customer's house.
In a rural environment, yeah, sure. Based on what I'm seeing in San Francisco, in an urban environment, you're going to be negotiating for roof space for many transceivers on many separate roofs. (I do absolutely agree that even that annoying tasks is certainly way less work than dealing with a local or state government that wants it to be impossible to run fiber through or along streets and sidewalks.)
Look, wireless service is almost guaranteed to be worse, but that has more to do with dodgy operators. The technology is fantastic, and when engineered correctly largely undetectable.
That said, in my time, I can count on one hand the number of installations where I was allowed to engineer the service correctly. And I can count on all the hands in a small city the number of times I have been called to rescue something extremely stupid, like shooting a link across a construction site.
>You can create a great wireless link. It still won't beat a great fiber link. I'm sure the best wireless links beat the worst fiber links, but...
When engineered correctly people tend to have absolutely no idea wireless is involved in their connection. Largely its a self inflicted branding issue. You see wireless being sold as "Fibre Extension" way too often for this reason.
Theres also factors that fibre people never consider, like mean time to restore a service. Even if you have a team of 24/7 engineers ready to mobilise, a fibre break will often take a significantly longer time to restore than a wireless outage.
Look at it this way as well: The slowest practical fiber connection is 10Gbps, individually per fiber. The fastest practical wireless connection is about 2.5Gbps, shared between all users.
Telephone and cable TV companies were explicitly envisioned as regulated monopolies in most places. Then it was cheaper for them to provide Internet over their existing lines than for a new company to come in.
I didn't think I've ever seen mention of a buyout in these articles. That could be something. Franchised ISP. Maybe Comcast is incapable of servicing an area effectively, so they could say something like "we'll give you x gbps of guaranteed throughout at the datacenter (or however it works) to our main line and teach you how to setup, you cover installation and maintenance". Just because it seems like it would've been easier for these guys to do only the installation and routine maintenance. But idk I guess they don't want to because they make their money anyway
So glad to see a renewed emphasis on proper wired infrastructure. It seems the "big boys" (Verizon, T-Mobile, etc) are heavily pushing wireless and not building out new wired areas, I assume because it's less capital intensive.
Hell if there's a way to invest in Prime-One, these guys seem to have their stuff together...
For anyone who doesn't know the area, Saline is adjacent to Ann Arbor, and along with Ypsilanti makes up a sort of greater Ann Arbor/UMich co-prosperity sphere. Saline is the kind of place you expect people to stand up a private fiber ISP; a place with an outer-ring suburb vibe, but far from any major metro, with lots of nerds.
If you ever have the chance to support a local ISP like this, do it! You can get some pretty sweet deals, the last time I had the opportunity to do this they threw in a /28 for "free" (agreed to two year terms)
When was the last time you had the opportunity to do this? 1998? I believe that what you're telling me is true, (and yeah, small ISPs are often really great to be customers of because of stuff like that) but -given the state of IPv4 allocations- I find it difficult to believe that it happened within the past ten, twenty years.
This is why national ISPs like Comcast have fought/lobbied tooth and nail to prevent municipal based ISPs from being created in various states (ie, Texas). Next logical step is starting a small ISP like these people have but they have the advantage of learning the skills and process (permitting with municipality) of doing this for other ISPs. There's also the capital aspect of this, which they apparently have.
> Comcast seems to have noticed, Herman said. "They've been calling our clients nonstop to try to come back to their service, offer them discounted rates for a five-year contract and so on," he said.
go figure. their monopoly/duopoly has ended, profits dropping like a rock in area, and now they want to compete.
Only billionaires and people fooled by Peter Thiel think competition is evil.
Wow the US really has it bad when it comes to home internet. In many European countries, you can get symmetric Gbit internet for 30-40 EUR (probably less in some places), and I haven't seen a data cap in forever.
The EU is better on average, but isn't universally great either. I pay 60 EUR for 200Mbit down/20Mbit up ADSL in Amsterdam, after my 6-month discount ran out. No fiber in my neighborhood yet. There's one gigabit provider in my neighborhood (Ziggo) and they have a bad reputation. For the same price I was getting FiOS gigabit in NYC.
I have 1.1Gb/100Mb on Ziggo in Amsterdam. I've had no real issues with them, at least for me the reputation is undeserved. I pay a bit more than that, but like €20 or so. They also give me a /57 or something which is nice (if a weird allocation, but I'm not going to use that many subnets anyway.)
I have had a fibre cable poking out of the footpath in front of my apartment for a year or two now, waiting for ODF or whoever to come and install it into the building.
Would it make you jealous if I tell you, that I get 10 Gbit symmetric fiber here in Switzerland (greater Zurich area) for roughly 80 USD/month with no data cap? And I can use my own router and could even go up to 25 Gbit if I want ;-)
Oh, and did I mention no CGNAT and it comes with a static /48 IPv6 net?
You are talking about init7, and they did have to fight tooth and nail and take it to court to get SwissCom to provide reasonable access to the fiber.
IIRC, Init7 explained that basically the internet wants to be fast, but ISPs want to spend lots of money on special equipment to slow down your internet so that they can sell speed tiers and data caps.
The US is much more captured that Switzerland, of course. Providers in the US get billions of dollars to expand their networks and provide service, and then don't provide the service, and nothing happens.
I'm paying under $40/mo for 10g symmetric here in southern Japan, that's including the extra charge for a public static IPv4 address (standard is a /56 IPv6 and tunneled CGNAT IPv4)
Here symmetric 4Gbit without a data cap (NL). Best of all, you can bring your own equipment. I have my Ubiquiti Gateway Max hooked up to fiber with a media converter (yes, the Gateway Max does PPPoE etc.).
My parents live in a small, countryside village. They have fiber at the same prices (including 4Gbit symmetric, though they are happy with a cheap 200Mbit subscription).
I don't think sonic has asymmetric internet anywhere.
It used to be symmetric gig, now they're deploying symmetric 10gig.
And the price is $49, although they just announced an increase to $59.
I had symmetric 10Gb for over a year, until they sent a new modem. When I tested the speed again, it was asymmetric. I figure it's the new modem, but the change may have been made before. I don't usually upload >1Gb, so I didn't notice when it actually happened.
If you haven't already, check to see if either Google Fiber or Monkeybrains is available in your area. Last I checked, the regs are still in place that prevent landlords from denying you access to an ISP of your choice.
Google Fiber isn't rolled out to most of the bay area. Monkeybrains is wireless with speeds significantly slower than what Comcast offers me. I've checked just about every wired ISP possible, and Comcast is the only option that services my neighborhood.
And FWIW, I own my house in the east bay — I am the landlady ;)
> Google Fiber isn't rolled out to most of the bay area.
I'm aware. And it's my understanding that -sadly- much of the Google "Fiber" deployment in the area is a WISP, just like Monkeybrains is. Quite a while back, Google Fiber bought Webpass and continued doing WISP deployment in the SFBA under the Google Fiber brand. (Because it's politically dreadfully hard to run fiber optic cables in the area.)
If you haven't contacted Monkeybrains for a minimum and expected speed quote at your site in a year or five, it's worth doing it again. It's my understanding that they aperiodically upgrade the hardware in their core network as well as the sort of hardware that they deploy at customer sites.
Monkeybrains' down-to 100/100 service is -on paper- far, far slower than the up-to 1400/40 service I was getting from Comcast, but the actual, delivered speed that I'm seeing from Monkeybrains varies between 300mbit and ~1000mbit (sustained) depending on what other folks are doing on their network. [0] I'm in a fifty-apartment building, so it's possible that they've installed faster gear on my roof than they install in smaller (or single-family) buildings. Reports on the Internet seem to be somewhat mixed, with some single-family buildings reporting ~1gbit service, and others reporting ~45mbit.
[0] Typical prime-time speed is something like 400mbit. Off-hours speed is frequently very close to 1gbit. The only time I've seen the minimum speed was when I had a poorly-crimped Ethernet cable between my router and the rest of my LAN that would intermittently only link up at 100mbit.
This "$80 for gigabit fiber, unlimited data" is a problem for me. I want to save money, I don't need a gigabit fiber but there's no other options. If there's $30 for 300Mbps unlimited data (I'm paying $40 for optimum) option, then I'll take it.
I wish these guys the best, but I've shifted more and more to Starlink. As long as their is a clear view of the sky, it works exceptionally well and is more than enough bandwidth. Plus, I can easily take it with me anywhere I go, which includes my campervan. This is great for when you're out in the middle of nowhere, with no reception, and you need access to maps.
I wish it was a bit cheaper, but someone has to fund that trip to Mars.
I've really been looking at some of the new mobile options for when I take the family camping but I still need to get some stuff done. With that said, over the last 30 days at home we've downloaded 4.8tb and uploaded 6.2 (no torrenting). I'm sure there are thing we could do different, but two people that WFH and do some semi data involved things... really just not sure how we could make that work full time.
I'm kind of surprised how expensive this seems. I always thought NZ ended up paying through the nose for the internet but these prices all seem roughly in line with NZ except they're in USD which is about 1:0.6 right now.
I once lived in a town with local high speed (although it was cable, not fiber). It really does make a world of difference in terms of what you pay and what kind of support you get.
It's disgusting that big telecom has been able to monopolize so much of the US for so long.
But kudos to them, that's the only way to break monopolies.
Although I do wonder how much they bet on people not using all that bandwidth based on their promises.
From everything I've ever read and my (admittedly old) ISP experience, people really don't use all the bandwidth. Yes, occasionally there are exceptions.
I understand the need for independent fiber ISPs. But are gigabit speeds really necessary? For me, a 300 Mbps connection is way more than enough for a four-person family.
Every bit per second that a WAN connection is slower than a CPU interconnect is too slow and that's only half hyperbole.
But I'm a realist so I'll take what I can get.
As your remote resources get faster and faster you start using them more like local resources, which can change (often for the better) how you do things.
If all you're doing is watching netflix then 25mbps per user is probably fine.
If you're working or creating or earning, then you want your connection to be as fast as possible, your distant end to be as fast as possible, and the hardware in between to be fast as possible.
> But are gigabit speeds really necessary? For me, a 300 Mbps connection is way more than enough for a four-person family.
I'm certain that one could make a sound argument that 300 Mbps is not necessary for that four-person family, and they could make do with a much slower connection. Back in the day, folks would be asking if it's necessary for your Internet connection to be always on. After all, it's no hassle at all to plug the modem into the house phone line and unplug it when you're done!
For me, switching from a 1400/40mbps cable connection to a symmetric-but-variable 300-1000mbps Ethernet connection meant that I was doing the same sorts of things, but often spending much less time waiting for them to complete. Related to that, it also made "content creation"-esque things [0] much, much easier.
[0] Which I'm declaring is a category that includes uploading and downloading huge files while working from home as a programmer/"DevOps" guy.
Gigabit is the slowest reasonable fiber speed. Even 10-gigabit is now very cheap to the point that some gigabit equipment is just 10-gigabit equipment operating at a low speed. They could artificially throttle you to even less, if it made business sense.
It's not a committed rate. Your individual line is a gigabit, but the upstream from your whole block is 10 gigabit so you can't all use it at once. Your guaranteed rate is probably have more like 20-50 Mbps, if that's what's confusing you. But it's extremely rare that everyone tries to use their gigabit all at once.
If it's a Passive Optical Network, you might be sharing a gigabit download with your block - you all share the same fiber - and you get substantially less than a gigabit upload due to the need for timeslotting. Gigabit PON is obsolete though, now you'd get at least 10G PON.
It's usually the upload speeds that are the bottleneck. When the upload is only 10 Mbps any video upload or large file upload will saturate the connection. In addition to having to wait for the video to finish uploading, if you don't manually rate limit the upload job speed or have anti-bufferbloat measures deployed any concurrent voice or video calls will suffer. During Covid shutdowns four people video chatting at once was no longer far fetched.
The scarcity of upload speed is why I learned how to setup OpenWrt's SQM feature, and upgraded to the highest package my ISPs offers.
Interesting. Yes, I guess the upload speed definitely matters, which cable/DSL don't provide.
I myself have fiber, and speeds are much less than 500 Mbps, but symmetric (with likely maximum two people using video calling at a time), so haven't faced that issue.
That's a bit more than I have (Starlink), but anytime a kid is downloading a big Steam game or big ISO file or something, everything else slows to a crawl. I also occassionally rsync large directories to/from cloud storage and that can also saturate. I've tried setting rules/priorities but it's a constant game of whack-a-mole
> I also occasionally rsync large directories to/from cloud storage and that can also saturate
Just offering some advice if you aren't aware. If you are, freely ignore. (And if you have advice in return I'd love to hear!)
For convenience, the rclone tool is nice for most cloud storage like google and stuff that make rsync annoying[0]
rsync also offers compression[1], and you might want to balance it depending if you want to be CPU bound or IO bound. You can pick the compression and level, with more options than just the `-z` flag. You can also increase speed by not doing the checksum, or by running without checksum and then running again later with. Or some intervaling like daily backups without and monthly you do checksums.
If you tar your files up first I have a function that is essentially `tar cf - "${@:2}" | xz -9 --threads $NTHREADS --verbose > "${1}"` which uses the maximum `xz` compression level. I like to heavily compress things upstream because it also makes downloads faster and decompression is much easier than compression. I usually prefer being compute bound.
Also, a systemd job is always nice and offers more flexibility than cron. It's what's helped me most with the wack-a-mole game. I like to do on calendar events (e.g. Daily, Weekly) and add a random delay. It's also nice that if the event was missed because the machine was off it'll run the job once the machine is back on (I usually make it wait at least 15 minutes after machine comes online).
Thank you! I do love rclone a great deal, and need to start using it more. I mostly use rsync right now because I've got some Contabo storage instances and ssh is already set up, so it's nice and easy, and I've already got the rsync commands burned into memory from decades of use :-) I also typically rsync with the same command to an external hard drive for an on-site backup.
Great tips! I'll definitely be using your tar command
Oh, that reminds me. When I had an Android (I already regret my iPhone) I used termux to write a very basic script to rsync data to machines and do so based on if connected to WiFi (with whitelisted SSIDs). Pretty easy to write and then schedule a cron job. Can be nice to put that onto other peoples devices and give them automated backups. Makes it FAR easier to do the 321 backup strategy. Also pretty easy to build a tracking app for a lost phone that way (like if haven't connected to home WiFi in x days send you an email). Both work really well when also using tailscale.
Let's take video streaming. I have a pretty compressed version of Arrival that's at 2GB and is a 4k movie ~2hrs long (the original file was ~2x the size). To stream that we need to do 2000Mb / (3600s * 2) = 277.8Mb/s. This also doesn't account for any buffering. This is one of my smaller 4k videos and more typical is going to be 3Gb-5Gb (e.g. Oppenheimer vs Children of Men). Arrival is pretty dark and a slow movie so great for compression.
Now, there's probably some trickery going on that can get better savings and you'll see used with things like degrading the quality. You could probably drop this down to 1.5Gb and have no major visual hits or you can do a variable streaming and drop this even more. On many screens you might not notice a huge difference between 1440 and 4k, and depending on the video, maybe even 1080p and 4k[0].
For comparison, I loaded up a 4k YouTube video (which uses vp9 encoding) and monitored the bandwidth. It is very spiky, but frequently jumped between 150kbps and 200Mbps. You could probably do 2 people on this. I think it'd get bogged down with 4 people. And remember, this is all highly variable. Games, downloads, and many other things can greatly impact all this. It also highly depends on the stability of your network connection. You're paying for *UP TO* 300Mbps, not a fixed rate of 300Mbps. Most people want a bit of headroom.
[0] Any person will 100% be able to differentiate 1080p and 4k when head to head, but in the wild? We're just too used to spotty connections and variable resolutions. It also depends on the screen you're viewing from, most importantly the screen size (e.g. phone).
First, if it was 2GB * 2 for the source of your recompressed copy, that's 4GB * 8 bits per byte = 32 Gigabits (Gb), or 32,000Mb. Two hours in seconds is 60 * 60 * 2 = 7,200 seconds.
32,000 / 7,200 is 4.444Mb/s. Streaming your 2 hour long 4GB movie could be done with ~5Mbit. A 1Gb/s connection could handle streaming ~200 of these movies.
Going back to Blu-rays as a source, an Ultra HD Blu-ray maxes out at 144Mbit but in reality most movies are encoded at a much lower bitrate. Most movies will cap out around 40-50Mbit. You could do 20 of these straight Blu-ray movies on a 1Gb connection.
> I have a pretty compressed version of Arrival that's at 2GB and is a 4k movie ~2hrs long (the original file was ~2x the size). To stream that we need to do 2000Mb / (3600s * 2) = 277.8Mb/s.
Your math is WAY off.
2 gigabytes / 2 hours is only about 2.22 megabits/sec.
> It is very spiky, but frequently jumped between 150kbps and 200Mbps. […] I think it’d get bogged down with 4 people
That’s just burst buffering as fast as it can, you didn’t capture the average. It doesn’t suggest it would slow down with 4 people. 4K on YouTube takes 20Mbps, so to parent’s point, you’ll have plenty of bandwidth to spare if 4 people do this at the same time on a 300Mbps line.
I mean the question is why limit it? I'm paying under $40/mo for 10 gigabit, bandwidth shouldn't be expensive, the expensive part is digging the last mile.
I have to admit, everytime I hear the "Two guys hated x, so they built their own!" I see the XCKD cartoon https://xkcd.com/927/
It's not a fair comparison; competition can drive price down, but I pessimistically just see two guys who'll inevitably join the Comcast billionaires club. That's just where these "small guys" end up.
I'm one of their customers. I often see that one green car parked down the road.
It's pretty good - their provided router is locked down to hell and they're on a cgnat, but not having to deal with Comcast's 1.2tb data cap is well worth it. Checking Comcast's site now, it seems that they now offer "unlimited" data. Interesting, that option wasn't there 6 months ago.
~100 customers seems too small for the amount of effort they have put in so far. They've been working along all the roads near me for about a year, and they're out there running fiber conduit every day. The houses out here are far apart. Hopefully, they can make it work.
> I'm one of their customers. It's pretty good - their provided router is locked down to hell and they're on a cgnat
This sounds like mine. I'm guessing yours doesn't support IPv6 because most fiber providers don't.
For the router, I already build firewalls so that. I pay $10/mo to escape their cgnat.
I've also alerted them to expect regular haranguing from me about deploying IPv6. Especially since bgp.he.net shows they have a /40 allocated to themselves; it doesn't seem to be used.
I've had less than 0.5% of customers ask for IPv6 from my fibre ISP. It's not worth supporting as a result. The main reason is that any service that is not widely used will have gremlins that result in poor customer experience, and if it's always the same handful of customers hitting problems or finding quirks, there is a real risk of poor word of mouth incident reporting that can harm the business. At least if something goes wrong with IPv4, it's going to be noticed very quickly.
Some people will say monitoring is all that you need, but I do not agree. There are a million different little issues that can and do occur on physical networks in the real world, and there's no way monitoring will have a 99% chance of detecting all of them. When incidents like the partial Microsoft network outage that hit certain peering points occurred, I had to route around the damage by tweaking route filtering on the core routers to prefer a transit connection that worked over the lower cost peering point. It's that kind of oddball issue that active users catch and report which does not happen for barely used services like IPv6.
> I've had less than 0.5% of customers ask for IPv6 from my fibre ISP
How many ask for IPv4? I understand your situation, it's a lot of work, for something that many won't notice. It's just that saying there's no demand because your average consumer, who also doesn't know what IPv4 is, isn't asking for it, is the mentality that keeps IPv6 from being implemented.
On the funnier side of things, we've also sometimes run into the opposite problem that we can't reproduce an issue, because it's only on IPv4 and 95% of the time everything we do is IPv6. But we're also not serving home users.
Static IPv4 addresses are closer to around 5% of customers. Nobody asks for IPv4, but some customers bring their existing or own wireless routers along and occasionally choose devices that are not IPv6 capable. Maybe in another 10 years those devices will finally be fully removed from service. The worst stragglers right now are the old combo DSL modems that effectively have no modern replacements -- it's just not worth spending money to replace them when customers are going to migrate to fibre soon enough.
I don't think GP meant static addressing but literally, how many ask for IPv4 service? None, because you just provide it; it's an unstated expectation.
Now apply that to IPv6 and you can see the point that (I think) GP is making.
Side note: The claim it is not widely used doesn't track. How many people use Google or Facebook? More than half of that traffic is over IPv6.
https://circleid.com/posts/ipv6-usage-in-the-u.s-surpasses-5...
I know that the GP wasn't asking about static IP addresses, however I was merely providing some statistics about what customer asks regarding IP addresses are actually seen on the provider side of things.
It's not widely used in my customer base as well as among a number of other small independent ISPs I know of. A significant number of the CPE we have deployed today do not support IPv6, therefore a significant number of our end users will not be using IPv6 resulting in only partial usage. That is what I am referring to: an incomplete deployment of IPv6 across a customer base without their knowledge of change and the end user's ability to identify issues will cause support issues which costs money and time.
Because of how IPv6 is provisioned, going from a no IPv6-deployed state to fully-deployed state will encounter transition issues. It's not provisioned the same way as IPv4, and that's the whole bloody problem!!! Sure, I can turn on handing out IPv6 addresses from our edge routers via PPPoE today, but it's not going to "Just Work" for all customers without CPE config changes (assuming the CPE even supports IPv6 in the first place for which a bunch of them do not!). It's a complete pain that has less than zero benefit for a small provider today since everything is available over IPv4 and nothing is IPv6 only. Heck, another ISP I know had a customer ask for IPv6 support, and then the customer didn't even bother to provision it on their own router!
Another example: take Ubiquiti's AirCube product line. It doesn't support IPv6. To deploy IPv6 it would be necessary to reconfigure things to make the ONU for the customer act as the router for the customer's home network and place the AirCube in bridged mode. That's assuming that the ONU even supports IPv6 yet as it did not when I looked into it on the order of 6 years ago. And yes, there are a few dozen customers on AirCubes that fall under this case. This means moving into a territory where multiple different CPE deployment configs are now required for less than zero benefit.
That there are gaps in IPv6 support has been a problem, is a problem, and will continue to be a problem. The industry isn't 100% IPv6, and I don't see how that is going to be the case in the next 5 years or even the next 10 years. Consumer gear isn't there yet. That was the problem 5 years ago, that was the problem 10 years ago, that was the problem 15 years, and that was the problem 20 years ago... What's different today that's any different than yesterday?
It's easy to deploy IPv6 in data centers where you control all the infrastructure on servers where operating system IPv6 support was mature decades ago. It's not so easy as a small ISP when dealing with pile of random consumer gear that an ISP's customers bring to the table where IPv6 support remains hit and miss.
So when you said `ask for IPv6` you meant `ask for a static IPv6 prefix` or something else similar to a static IPv4 address? Or is this an apples to oranges comparison?
And then you say `Nobody asks for IPv4` - so nobody asks for IPv4 and 0.5% ask for IPv6?
2 of my customers explicitly asked for IPv6 prefixes. Another ISP had a couple of users actively using IPv6 15 years ago, but it bitrotted and nobody cared enough to ask for it to be fixed. Another user recently asked to be provided with IPv6 (actually, forcefully demanded it rater) and then didn't even bother to enable IPv6 on their router.
So, yeah, I don't see IPv6 being relevant as a small ISP today.
I have had a grand total of one customer ask in almost 10 years. I have the block and announce it but there are so many dangers with implementing it that I am scared to even try yet until we slow down a little.
> I've had less than 0.5% of customers ask for IPv6 from my fibre ISP. It's not worth supporting as a result.
Big, evil, hated Comcast has full ipv6, and I doubt any of its customers asked for it either. Instead people complain they’re only getting a /60.
Comcast was forced to go to IPv6 because they ran out of IPv4 addresses in the private address space to use for management of their network (think of how each and every cable modem needs a an address for management in addition to all the routers and CMTSes). I was a fly on the way inside one of the router vendors when this took place more than 15 years ago.
If you already have to do CGNAT, why not IPv6 as your core network with NAT64 at the border and 464XLAT on the CPE? It gives you best of both worlds.
I'm not doing CGNAT. We were able to get enough IPv4 addresses directly from ARIN a few years ago after being on the waiting list for a couple of years. It's a pity that widespread fraud depleted that pool faster than it should have been.
CPE support for IPv6 has generally been garbage with it taking 15-20 years before the bare minimum was supported by mainstream router vendors. Even today there are still vendors that assume only IPv4 support. In my opinion the IETF really screwed up when they made IPv6 more complicated than just IPv4 with more address bits. The incumbent in my area generally uses PPPoE in their access network, but routers that supported PPPoE and prefix delegation basically didn't exist in 2010, and only started being available circa 2015 (in part due to the required bits not existing in OpenWRT and the hardware vendors' software development kits for their chipsets). Sure, we're 10 years further on now, but there remain a number of vendors that only support IPv4 for management of devices (cough Ubiquiti cough) in parts of their product line.
That said, there are features of IPv6 that are absolutely awesome for carriers. The next header feature that pretty much eliminates the need for MPLS in an IPv6 transport network is one such item that makes building transport networks so much cleaner when using IPv6 than IPv4. No more header insertion or rewriting, just update one field and fix up the delta on the checksum and CRC. They just aren't really applicable for smaller networks.
Ah, okay, well if you already have the IPv4 address space available for all your customers that's a different story. I can understand why you want to wait to dual-stack in that case.
I do think NAT64/464XLAT is a pretty good architecture for new ISPs that can't get their hands on IPv4 space, though. Or even MAP-T, but CPE support isn't really there yet.
CPE support is the issue and has always been the issue! People assume that IPv6 is done, but it really isn't in the consumer networking gear product space.
Android and iOS have pressure on app developers to support IPv6 or at the very least function on 464XLAT. On home broadband people could connect anything including programs that use literal IPv4 addresses and break on 464XLAT. Things like corporate VPNs, Skype, online gaming. Ironically these services would benefit the most from supporting IPv6 but they've evolved to deal with IPv4 NAT to the extent that they've become dependent on IPv4 connectivity.
Great: your client device supports IPv6 while your consumer networking gear does not. How's that going to work exactly?
Most people just use what their ISP issues. Comcast supports IPv6 and so does their CPE.
For me, no IPv6 = no business. I don't think it's acceptable to build a network on IPv4 only at this point, it speaks to being willing to cut corners and not do things the right way just because it's easier.
I worked in this space for a while, in the US. Outside of the major cities, Internet service falls off extremely quickly. Like, shockingly so: you can be as close as fifty miles from, say Philly or Flagstaff and have zero fiber, zero cell coverage, just nothing.
The people who attempt to fill these gaps are commonly rural telephone companies, electric cooperatives, tribal entities, or mom and pop shops where the owner grew up on a Ditch Witch and only knows as much IP networking as essential to light up the fiber and get the packets flowing upstream.
They are enormously resource constrained in ways you might not expect, too, eg operations can grind to a halt because everyone is out with a chainsaw after a storm, or because the Guy that Knew Stuff about their network died suddenly.
They are very, very unlikely to decide to run an IPv6 network just because. There's no upside that makes the juice worth the squeeze for them.
In principle, IPv6 core networks can actually be very beneficial for small providers just starting out if they're not able to get IPv4 addresses for their customers and are forced to use CGNAT.
In an IPv4-only CGNAT setup, all the traffic has to flow through the CGNAT gateways, and that gear is stupidly expensive. Having IPv6 in the mix means that anything that supports IPv6 (such as most streaming services) won't hit the CGNAT gateway and can just be routed natively. This can really save money on CGNAT hardware.
For implementation, you can use NAT64/DNS64 for your CGNAT setup and implement 464XLAT on the CPE. This keeps your whole edge network IPv6-only so you don't have the complication of maintaining two parallel configurations on the edge.
There is also MAP-T, which is even lighter on infrastructure since it pushes all the state into the CPE and avoids the complication of stateful CGNAT. But unfortunately CPE support for it is pretty limited at the moment.
AFAIK in Germany for DOCSIS it's common to have DS-Lite in the mode with lightweight 4over6, so the CPE does the NAT and the carrier just range checks the client side IPv4 port to fall within that customer's block.
No state at the carrier; IP:6 native network.
If you buy a full IPv4 you get one for yourself but I expect still through the same tunneling machinery to keep the network IPv6-only.
> Outside of the major cities, Internet service falls off extremely quickly.
Saline is less than 10 miles from Ann Arbor.
> The people who attempt to fill these gaps are commonly rural telephone companies, electric cooperatives, tribal entities, or mom and pop shops...
That's fair, but at some point, you need to recognize you are competing with a major ISP. No one appreciates it when you come in, tear up the roads, and then pull out once the incumbent ISPs bump up their speeds ever so slightly. (Looking at you, Google.)
> They are very, very unlikely to decide to run an IPv6 network just because.
No one deploys IPv6 "just because," and yet more than half of the traffic to major sites is IPv6.
> Outside of the major cities, Internet service falls off extremely quickly
I live half an hour from a state capital and my only option is cable... the coaxial cable they laid bare on my flower bed decades ago. I dig it up about every other year when planting. It's not even in a conduit!
I wish I could say no IPv6 no business. There are only 2 ISPs here, one cable and one fiber. Neither have IPv6, the smaller ISP also does CGNAT because IPs are expensive. I'm trying to convince them that they could save money with less powerful CGNAT hardware if they deploy dual stack.
I agree in principal but if the only other option is Charter/Spectrum/Comcast, you bet I'm going with the "lazy" person's fiber.
I have spent most of my career under the thumb of fucking cable and I'd sooner slam a car door on my nuts than go back to paying so much money for such garbage service.
"I'm guessing yours doesn't support IPv6 because most fiber providers don't."
Yeah, what's up with that? I just got switched on to fiber and the CGNAT for IPv4 doesn't shock me much, but what's with the no IPv6 in 2025?
I know enough to deal with it, but what's the deal? Is there something systematic here?
Everybody can muddle along without IPv6, so it's easy to make it a very low priority. Especially for small shops that are struggling just to create a viable business. IPv6 needs something more to motivate it, a web destination or application that is only available on IPv6.
> IPv6 needs something more to motivate it, a web destination or application that is only available on IPv6.
How about not having to pay for (as) beefy CG-NAT hardware because people that go to Youtube, Netflix, MetaFace, TikTok, etc, can directly connect via IPv6.
Hadn't thought of that, but it might not be a huge savings unless you were to go ipv6 only. If you're still going to support ipv4 anyway, the hardware savings might not be too significant.
> Hadn't thought of that, but it might not be a huge savings unless you were to go ipv6 only.
Even a small number of devices/services not supporting IPv6 can have huge costs:
> Our [American Indian] tribal network started out IPv6, but soon learned we had to somehow support IPv4 only traffic. It took almost 11 months in order to get a small amount of IPv4 addresses allocated for this use. In fact there were only enough addresses to cover maybe 1% of population. So we were forced to create a very expensive proxy/translation server in order to support this traffic.
> We learned a very expensive lesson. 71% of the IPv4 traffic we were supporting was from ROKU devices. 9% coming from DishNetwork & DirectTV satellite tuners, 11% from HomeSecurity cameras and systems, and remaining 9% we replaced extremely outdated Point of Sale(POS) equipment. So we cut ROKU some slack three years ago by spending a little over $300k just to support their devices.
* https://community.roku.com/t5/Features-settings-updates/It-s...
* Discussion: https://news.ycombinator.com/item?id=35047624
Sadly the post is now behind a login: what happened later was Apple donate a bunch of Apple TV devices to the tribal ISP and that cut their IPv4 usage by an order of magnitude (or some ridiculous number) and there were major savings. The ISP then recommended AppleTV to all of their customers to get the best viewing experience (because of the latency/overhead of CG-NAT when streaming video).
So the more you move over the more headroom you have for the broken IPv4-only systems. AIUI, the rollout of MAP-T/E has helped in that things are more stateless, and more work is done at the CPE, but there's still overhead.
We used to have freeipv6porn.com, lol. But I suspect that was a joke as much as anything else given how much porn you can get for free all over the Internet.
The eggs need some chickens first.
Thankfully, they are doing IPv6, although one day I had some weird issue where IPv6 was broken but if I disabled it ipv4 was still working. Could have been my fault, IPv6 is generally new to me (not much of a network person).
I get the impression that they are still learning to run an ISP, both technically and customer facingly. It's weird - I learned more about them from this article than from actually being living here with them.
Surprised they aren't deploying NAT64/DNS64 with 464XLAT on the CPE. You get essentially the same setup as CGNAT for IPv4 services but your whole core network is native IPv6 so you only have one set of address space to manage and your customers will be able to directly connect to anything IPv6 related.
How would you as a customer tell if they were?
Because you'd have native IPv6
Comcast has pretty good IPv6 support
Charter does too except their allocation size is your puzzle to figure out. Maybe it's /56, maybe it's /60, maybe neither of those will work. I've run into all the above.
since tailscale exists, why would you care about cgnat or even pay to escape it?
I'm not the only person connecting to my machines.
Some applications want to open ports and don't have the server-side infrastructure to punch a hole through NAT. Especially P2P apps and some games.
Sometimes I want to run a small, low-traffic web server from home.
Sometimes I'm connecting to my network from a machine that I don't control and can't install Tailscale on.
Tailscale uses the same range as CGNAT.
https://github.com/tailscale/tailscale/issues/12829
You shouldn't be seeing the CGNAT addresses inside your home network, should you?
Depends on your network, router configuration. But in any case, if you install it on a router where the upstream uses CGNAT yes, you would.
Router straddles inside and outside. I know there's a way where you could put tailscale on your router but it's not the normal way to use it.
> Checking Comcast's site now, it seems that they now offer "unlimited" data. Interesting, that option wasn't there 6 months ago.
It's been there since they announced the data cap. I thought the unlimited bundled with leasing their higher end hardware came first, but the email from 2016 announcing that our plan was getting the cap mentions being able to pay for unlimited.
You've always had the _option_ of paying extra for unlimited data, however its only in the past month or two that they've started offering unlimited data as standard (in select markets).
https://arstechnica.com/tech-policy/2025/06/stung-by-custome...
I’m sorry I still don’t get it. Could you explain that in different phrasing ?
A comcast customer always had the option to pay for unlimited data. I get that part. What is the 2nd part? “Started offering it as standard” means what?
In markets where Comcast has actual real competition, they "include" the unlimited data (aka no cap) with no extra charge when you sign up for their gigabit plans.
You pay for their highest tiered and highest bandwidth plan and they have the audacity to impose a cap making that bandwidth work against you? Crazy. My household internet usage is quite modest, nothing anyone on HN would call data intensive—casual video streaming being the lion's share, and I blow out 1TB every month without fail.
In the Bay Area Comcast offered (2017-2023 at least) internet with a default 1-1.2TiB/mo data cap that you can lift for the month for an extra 10-20usd (I don't recall, my roommate who played CoD was the one paying for this by himself on every month with huge updates).
There's barely any competition here. You can pretty much chose from Comcast Business or XFinity, which both are just Comcast because of a free market with free as in not in jail.
Really 10-20 more? When I asked, and I'm in the Bay Area, the unlimited plan was $5 a month more than it would cost if I leased their modem.
How much more was it than if you weren't leasing their modem?
It used to be 30/month for me. I was not renting their modem and got charged more for it.
If starlink ever gets more capacity, I'll probably switch. Right now I think the only way to get gigabit down on starlink is with four or five accounts and manually bonding the dishes together. As soon as that obstacle goes away, Comcast will have competition in my area and I intend take advantage of that.
That's not true. I tried getting unlimited data like 7-8 years ago and they said I needed a business account to get it.
What ISPs offer and how much they offer it for tends to vary wildly region to region.
If you live in a region where they have no meaningful competition (which is still fairly common in a lot of places in the US) well bend over and lube up.
They'll vary wildly, as much as they think they can get away with, in the hopes that you will never use the service and pay them as much as possible for it, and they'll bury your mailbox with crap to try to wear you down into coming back.
They will happily let you pay for years, for services that no longer exist, no longer connected to any of their networks. They'll take you to court rather than pay anything back; they know they are receiving extra money, and there's a significant amount that comes in, but "oh, it's so confusing, and there are so many legacy systems, we can't possibly catch every mistake."
The money they shuffle back and forth between each other, daily, reeks of book cooking - you might have a stretch of 20 miles of trunk in which there are 20 separate owners - not concurrent riding separate fiber lines, but in sequence, each paying rent to or getting rent paid by the adjacent rider, even though only a single company actually services the entire span.
It's funny how construction companies and ISPs get these rackets going, and then when people come along like these PrimeOne guys and offer a reasonable rate on a decent product, it's somehow vastly disruptive and threatening.
They'll expand, and be encouraged and allowed to expand, and after 5 or 6 years, the big ISPs will start circling, and eventually buy them out, and they'll retire happy. AT&T or Lumen will own their network inside of 10 years, and they'll claim it's modernized and upgraded infrastructure. People with shitty oversold undermaintained cable internet will be left alone until the money stops.
Starlink to phones is great, if it only didn't make ISPs so much money handling the base stations on the ground.
There's fiber all over the US just hanging there, unused, unmaintained, because merger after merger after merger left giant piles of assets under the ownership of companies like comcast and centurylink and at&t, who left infrastructure to rot, often built with public funding, and maintained their local monopolies and shitty service.
Whatever it is we're doing to regulate the industry at a federal level isn't working, but I imagine that's where a lot of the money goes.
A healthcare company I used to work for had a location in CA that, on paper, had 3 circuits - AT&T, Verizon, and Comcast. Each had their own Ciena box in their telecom closet. They were getting a circuit upgraded so one of them was newly installed and we needed to cut over the slowest one to an upgraded one.
Working with the director there, IIRC we traced down the Verizon and Comcast box as actually being connected to the AT&T box. After some checking of circuit IDs on boxes it seemed that the LEC for both the Verizon and Comcast circuits was AT&T, and AT&T was the actual owner of that single physical fiber going out there.
I wish I could sell the same thing 3 times, lol. Contracts were signed on all these circuits so there was no getting out of it.
Yeah they’ve offered unlimited for a few years now for an additional charge. Now as of last month all plans are unlimited if you just update to them. It wound up being cheaper for me too.
However I still applaud these guys. There needs to be more competition.
Comcast similarly removed their 1.2TB cap in my neighborhood within months of us getting fiber. It's almost like the only reason for the cap was because they could get away with it when there wasn't any competition.
Comcast is notorious for exploiting places that don't have any other real options. Just before Google Fiber was activated in my area, Comcast stepped up their game big time. The only problem is that they had spent years nickel and diming me for actual connection speeds that didn't even come close to their advertised rates, and their latency/jitter is garbage compared to fiber. Comcast clearly doesn't want to have to compete. In their defense, their connection was rarely down.
When I lived in downtown Oakland CA, Comcast literally could not keep up price-wise with the competition. Their customer service jaw would drop when I told them our local fiber offered a flat fee cheaper than theirs for 10 gigabit symmetrical fiber. On top of that there was another local microwave wireless option that wasn't too terrible.
The only thing in the end their salespeople could do was offer TV bundles but still wasn't cost-competitive. Not sure what their offerings are now but it was such an easy decision to switch.
> is notorious for exploiting places that don't have any other real options.
Isn’t this standard competitive practice ? Charge what the market will bear.
I don’t know if I’d call that “exploitation”. If there’s one gas station 90 miles from every other gas station in the Nevada desert, they’re gonna charge more, aren’t they?
Yes, it certainly is. But isn't it interesting that Comcast is almost universally hated? I used the word "exploit" simply because had they treated their customers better and focused on putting their best foot forward, I don't think they would have bled customers nearly as quickly.
I live on a dead end road with two ISPs and the one that isn't comcast is unusable. Comcast pretty much laughed at me when I asked if there was anything that could be done to drop my overpriced bill
That’s exactly it and they admitted it last week.
https://arstechnica.com/tech-policy/2025/06/stung-by-custome...
> the only reason for the cap
Correct. It was a very calculated decision. They were squeezing out profits by trying to move heavy users to the next tier of service. But this only works if they have a monopoly.
Feature-wise it doesn't matter because you're still going to have to play the price haggling game. Other providers don't renegotiate every 6 months like they do. They have more in common with Waste Management than with a respectable ISP.
I'm on the other side of the country and was a Cox customer for over a decade until they decided to add a data cap to their plans. Fortunately, wyyred rolled into town right around the same time, offering fiber at higher speeds, no data caps, and half the cost. It was an easy decision. I also noticed that Cox is now advertising unlimited data for free. Too little too late.
their provided router is locked down to hell and they're on a cgnat
So not actually better than Comcast, just bad in a different way.
I love how these guys are trying to sort the lack of competition in broadband, stood up and did something about it and all the geeks on HN are upset that they are not doing something that the majority of their customers would not give shit about or even come close to understanding.
Everyone here that has started a company to challenge the entrenched monopoly raise your hands please.
I understand the tech deeply and that does not translate to the practical needs of trying to run a successful business.
I raise a toast to these guys. Well done.
Given they have a hundred ish customers its likely at most a couple would even know how to do that, or care. It's just not worth their time to worry about this right now. They seem to be doing just fine
> their provided router is locked down to hell
From the article, it sounds like the "default" option is for the customer to supply their own router, which I appreciate:
> Prime-One provides a modem and the ONT, plus a Wi-Fi router if the customer prefers not to use their own router.
Modem and ONT? I'm under the impression that there's nothing called a "modem" for fiber, and that the ONT serves a similar role. Am I confused?
No, that's my understanding as well.
My fiber installer referred to the Adtran 632V ONT he installed as the "modem".
He installed two other junction boxes (one outside the house near/under where the fiber attaches to the wall of the house, one inside near the ONT) but they're just passive optical couplers allowing them to swap out fiber segments in the event of fiber damage without re-running the entire install.
Can’t speak to this exact circumstance, but more generally: The ONT translates the SFP+ networking to fibre optic, but the modem is still somewhat necessary for logins if you use PPPoE as a wrapper for example. In telecom fibre optic, it often also assigns a particular vlan to internet packets and separate vlans for TV and phone. But I’m not an expert here, just explaining why I needed a modem function in my router as well as a media converter to house the ONT.
As far as I know, nobody uses separate boxes for the modem and router, that kind of thinking died when wifi became more widespread and included by default with ISP plans.
I wouldn't really call that a "modem" though, it's not really doing modulation/demodulation work to convert between media types. The terminology I usually hear for the provider's box handling any final authentication and VLAN splitting is usually a "residential gateway", which can be configured to bridge to a client's equipment.
Definitely splitting hairs here though on terminology.
The component that does the PPPoE and VoIP VLAN is typically just referred to as a router (or a "residential gateway" for companies that want to sound less technical), I've never heard as it referred to as a modem, usually the ONT is referred to as a modem as it's MODulating/DEModulating the optical signal.
It was there 6 months ago, because when I moved and had to switch to comcast in 2021 I found out about the cap after ~5TB/mo
I vaguely remember reading something about their consolidating plans and simplifying pricing slightly. Part of that was eliminating the data cap.
This article couldn't have passed through my inbox more than 6 weeks or so ago so it is a very recent change.
"Everything that we're doing is all underground."
This indicates that their local and state governments aren't (at this time) captured by the incumbent cable provider.
A captured state gov will pass laws to thwart new infra deployment, commonly written by ISP interests. A captured local gov will never approve deployment or slow-walk permitting in an attempt to bankrupt the upstart.
more explainers: New suburban fiber infrastructure means either trenching or pole hanging. The local gov issues permits for both but poles also require the cooperation of the pole owners. This last adds the PSC to the mix.
Recalcitrant pole owners are known to stall and kill infrastructure deployment - especially where going underground isn't an option. Some PSCs mandate that pole owners cooperate. Some PSCs abdicate that responsibility and are examples of regulatory capture.
I’ve been hearing about “captured government” with respect to fiber deployment for two decades now and the folks on that soap box have made absolutely zero progress on improving deployment of fiber infrastructure in that time. Tilting at that windmill isn’t working, because for the most part that’s not the real problem.
Why isn’t the Bay Area a hot bed of fiber deployment? You think Comcast in Philly has more pull with Cupertino and Mountain View than Google and Apple? No! Internet in the Bay Area is shit for the same reason all the infrastructure in the Bay Area is shit. The government makes it slow and difficult to build anything.
Comcast installed fiber to my house back in 2018 or so. The permitting took months. And this was to run Comcast fiber on poles where Comcast already had their own cable lines. And my county is actually pretty efficient with permitting. It’s just that American municipalities absolutely hate it when anyone builds anything.
I guess Comcast doesn't need to capture the local government in places where it's already illegal to build anything. But in other places it has definitely happened.
In most places, permitting is the most direct and immediate roadblock. If you don't start there, you won't make any progress on the issue.
I live in a blue state that actively encourages municipal and cooperative fiber deployment: https://mdbc.us. It's had approximately zero impact outside some rural parts of the state.
Well, "deploying fiber" does not equal "starting a residential FTTP ISP". A ton of fiber infrastructure is only offered B2B.
If you think internet in the bay area is shit then you haven't seen how bad it can get. Even other large cities within California like LA and SD are worse.
Having lived in all three mentioned areas, none seemed particularly bad unless your standard is fiber straight to home. And seemed like Bay Area had more of that if anything.
I am absolutely not defending Comcast here but its worth pointing out that every anecdotal bit of evidence about performance and reliability can be true.
Network performance in that last mile can differ by block and even by season. An otherwise functional run of coax might have intermittent ingress but only shortly after it's rained while cold out.
This isn't even counting all of the flaky performance anecdotes that really boil down to overcrowded Wi-Fi or poorly configured consumer gear or anything else that isn't strictly the fault or problem of Comcast.
In college, we had the combination of spotty wifi + 10 dudes concurrently torrenting Shrek 2 + jank Android tablets running rogue DHCP servers and taking router's IP + rats chewing through cables, yet every time the internet went down, they called Comcast. And it was never actually Comcast's fault.
Maybe it's good enough that not very many people care. I moved around San Jose, Mountain View, Berkeley, and Sunnyvale, never noticed problems with Comcast or AT&T. Was expecting flakiness after hearing all these bad stories, but no, it was reliable.
What you don't get often is fiber-to-home, or great upload speeds. But most people aren't running big home servers.
CGNAT isn't a matter of cheapness. There are more Internet users than IP addresses. Someone has to draw the short straw, and that's the providers that came late to the party.
Now, if they're doing CGNAT and also don't have IPv6, that's just crap. They shouldn't be allowed to call themselves an ISP at that point.
Sorry for ninja edit, yeah I called CGNAT cheap because there are still ways to purchase IPv4 addresses, with caveats.
This was even a major hurdle for Google Fiber. The incumbent ISPs did everything they could to obstruct them from installing fiber, and it was fairly effective even against someone with deep pockets like Google.
In fairness, google tried to bypass all of the systems that already exist and are widely used for doing pole attachments.
Looks like they're somewhat rural which probably makes it way easier. I was a project manager for a Telco years ago and the process to get fiber run in an established city is crazy. Had no idea how much was going on under the roads until I had to plan out conduit boring projects.
It's not that easy. Poles vs trenches are a tradeoff discussion. FWIW I was once in construction digging trenches and I'm German, so I might be biased a bit.
Pro poles / open air:
- very, VERY cheap and fast to build out with GPON. That's how you got 1/1 GBit fiber in some piss poor village in the rural ditches of Romania.
- easy to get access when you need to do maintenance
Con poles / open air:
- it looks fucking ugly. Many a nice photo from Romania got some sort of half assed fiber cable on it.
- it's easy for drunk drivers, vandals (for the Americans: idiots shooting birds that rest on aboveground lines [1][2]), sabotage agents or moronic cable thieves to access and damage infrastructure
Pro trench digging:
- it's incredibly resilient. To take out electricity and power, you need a natural disaster at the scale of the infamous Ahrtal floods that ripped through bridges carrying cables and outright submerged and thus ruined district distribution networking rooms, but even the heaviest hailstorm doesn't give a fuck about cable that's buried. Drunk drivers are no concern, and so are cable thieves or terrorists.
- it looks way better, especially when local governments go and re-surface the roads afterwards
Cons trench digging:
- it's expensive, machinery and qualified staff are rare
- you usually need lots more bureaucracy with permits, traffic planning or what not else that's needed to dig a trench
- when something does happen below ground, it can be ... challenging to access the fault.
- in urban or even moderately settled areas, space below ground can be absurdly congested with existing infrastructure that necessitates a lot of manual excavation instead of machinery. Gas, water, sewers, long decommissioned pipe postal service lines, subways, low voltage power, high voltage power, other fiber providers, cable TV...
[1] https://www.usgs.gov/news/national-news-release/illegal-shoo...
[2] https://ucs.net/node/513
There's a huge downside to poles where I'm based: permit shenanigans by pole owners that delay projects and allow incumbents to destroy competitors. Granted, some municipalities do the same thing. One local municipality I have to deal with responds to permit requests almost instantly, while another takes weeks of pestering to acknowledge even the most basic of permit requests.
For anyone starting out today, I would strongly recommend having a planned legal / regulatory strategy to fall back on in the event that excessive delays occur by parties you cannot avoid dealing with.
More importantly, if they go on poles Comcast can "accidentally" cut their lines all the time.
Bell techs have done that and caused outages for customers a number of times. One outage ended up costing close to CAD$7900 to repair that they settled for due to it occurring on a weekend in a rural area during a snowstorm that required third party traffic control to meet the Book 7 requirements (on a weekend at the bottom of a hill in a snowstorm with insufficient visibility for us to work safely while doing a road crossing). Normally that would only be a $1k-2k or so, but traffic control, lack of slack due to the location of the cut and overtime for all parties involved really drove costs up. Less than 30 seconds with a pair of wire cutters can be downright brutal to repair if the conditions hit the worst case.
Meh, here in Germany you got the same issue with trenches. It takes ages to coordinate digging them, I think the worst example simmered for two years until the permits arrived. And then, it's a nightmare because you can't just cut off people's courtyards and parking spots for any time longer than absolutely required, so as soon as you're at depth you gotta cover the trench with steel plates so cars and pedestrians can cross...
The fiber installing crews around here go street by street and usually do one street section per day/or two.
Germany also does a lot more construction on main roads at night compared to what we see in North America.
>Poles vs trenches are a tradeoff discussion. FWIW I was once in construction digging trenches and I'm German, so I might be biased a bit.
when i got this far I literally thought you were making a joke about Poland.
> […] or moronic cable thieves to access and damage infrastructure
Manufacturers should print on the sleeve "fibre optic only. no copper". :)
> [Buried pros:] Drunk drivers are no concern, and so are cable thieves or terrorists.
Except for that one old lady who took the country of Armenia off the air:
* https://www.theguardian.com/world/2011/apr/06/georgian-woman...
> Manufacturers should print on the sleeve "fibre optic only. no copper". :)
Doesn't work, sadly. They just DGAF, cut, rip and go.
those days it's not trench digging (unless it's next to highway with machine that in one pass will trench and lay conduit/cable), it's trench drilling with something like this https://www.ditchwitch.com/directional-drills/
frontier installed fiber in my area using this method. relatively quick and no damage that needs to be "aggressively" paved over.
Can't do that in a century old city street with century old utilities in it. They dug up the sidewalk with shovels here as a result; though using machinery to move the soil they dug up and handle the asphalt that's covering the sidewalk along part of the street.
Yeah you can directionally drill across a front yard typically; but actual urban areas are just too filled with infrastructure.
before drilling there was a team that mapped and marked all underground utilities. some of the drilling went between front yards (this is where underground vaults are/were), but a bunch was done below sidewalks and across the streets.
I am in a rural area of Texas and I just recently got access to fiber. The other competition is ADSL and DOCSIS providers - AT&T and Optimum.
Optimum had their entire service area bought out by Comcast the day after I switched. Comcast has since broken every major utility at least twice and my fiber connection three times by working on the old infrastructure. I think Optimum won that trade. I can't imagine many residents are going to prefer Comcast over $80/m for no-bullshit internet, especially after the water main break they caused last week.
These FTTP providers have the game solved in Texas. I've seen them do 500-1000 homes in <30 days. Their directional drilling expertise and aggressive neglect for 811 seem to get things done very quickly. There are some areas with competing fiber providers now. I've got 5gbps symmetric for $110/m and I live in the woods. Trees go through power lines and the fiber infra is completely unaffected. The only utility left to bury is the electricity, and they're actively working on that in some areas now.
> aggressive neglect for 811
Texas regulations are quite something. My friend told me that the closest thing to regulation and zoning they have is... an HOA. This was the first time in my life I heard anything positive about HOAs.
Still waiting for someone to do the same in Bay Area. Many parts of it don’t have any fiber optics options, even though Sonic does provide some in the north.
AT&T put an optic cable at my curb 10 years ago (most likely due to imminent competition from Google Fiber internet), but then never lit it (most likely because Google dropped their effort due to complications with cities)…
Sonic is doing this in sfba. Used to be att reseller now they lay their own fiber, 50% cheaper plans, byo router, ipv6 that actually works, great service.
Pardon my ignorance but what is the benefit to ipv6 for local, consumer internet?
Online gaming: No "strict NAT" program. No conflicts with having multiple consoles at one locations. No port forwarding needed.
VPNs: Business IT usually uses some weird IP range like 172.xxx.xxx.xxx so avoid conflicting with the popular 192.168.1.xxx or 10.xx.yy.zz. When two companies merge there is now an IP range overlap and a renumbering has to be done.
P2P file transfers: No port forwarding needed.
Self-hosted servers: No port forwarding needed.
Video chat/VoIP: One reason video chat still suffers from bad quality is that video is proxied through cloud servers to deal with NAT. With more IPv6 video chat services can use more direct connectivity lowering costs and improving quality.
Avoiding CGNAT. If anyone on the same CGNAT group in your ISP got banned you will also be banned. Many Internet services don't have IPv6 and they often cite IPv4 based reputation as the reason why they won't deploy IPv6.
> When two companies merge there is now an IP range overlap and a renumbering has to be done.
That has nothing to do with OP's question about consumer internet concerns. However, I've been through several corporate acquisitions. The thing that took the longest to work out? Getting the new company's name on the paychecks.
The thing that took the second longest to work out? Renumbering networks and consolidating and decommissioning hardware for which there just wasn't enough RFC 1918 space available. Every. Damn. Time. this process dragged on and on and on, when IT could have saved 6->18+ months of future work [0] by generating a /48 ULA and using that for every internal thing but the stuff that was IPv4-only.
[0] ...and god only knows how many dollars in labor costs...
The big customer-visible feature is that every machine on their LAN can be globally reachable. For ISPs that absolutely cannot get enough IPv4 space to give their customers even a single globally-reachable IPv4 address, this would be the only way to get any global IP space. IME, edge routers that are intended for networking noobs to use often set up their firewalls so that they block inbound unsolicited IPv6 traffic, but (unlike with IPv4 NAT) there's the option of opening up multiple hosts to some or all inbound traffic.
Another feature that I find to be pretty stupid (but that some folks seem to really like) are IPv6 "privacy" addresses. Because each host usually is assigned an IPv6 address in a subnet that's 64 bits wide, most mainstream OS's have configured their IPv6 address autoconfigurator to set one stable, "permanent" address, and to set a parade of periodically changing "temporary" addresses. The OS is usually configured to prefer the permanent address when software asks for a socket to listen on (and sockets that handle replies to that listen socket), and those temporary addresses are preferred for sockets that initiate outbound traffic. The idea is that this is supposed to confuse tracking, but I'm very skeptical of its efficacy in the real world.
Finally, a customer can also usually get enough IP space to make globally-reachable subnets on their LAN. Depending how the ISP has configured things, a customer can get between four and 256 subnets. These subnets are handy to provide networks that provide globally-reachable IP addresses, but that can be easily logically isolated from the rest of the LAN by the router.
My feeling with privacy extensions is that it's more to prevent enumeration of a network by looking at outgoing traffic. It may have a degree of tracking protection too, but that's probably less important, and is about equivalent to everything being behind a single NAT address.
The RFC 4941 problem statement [0] notes that an attacker who can look at outgoing traffic from a network will not be thwarted by "privacy" addresses. The background section of that document goes on to say that "privacy" addresses do little-to-nothing to thwart correlation attempts against client networks that have few hosts on them.
IMO, something like what's described in RFC 7217 [1] (changing the interface identifier used for "permanent" addresses from the interface's MAC address to something that mixes in the advertised prefix) is a much better way to address the concerns described in section 2.3 of RFC 4941.
[0] <https://datatracker.ietf.org/doc/html/rfc4941#section-1.2>
[1] <https://datatracker.ietf.org/doc/html/rfc7217#section-4>
For regular folks there isn't much benefit tbh. Mainly I think it simplifies ISP architecture and offers slightly faster (like 10%) performance but ISPs have to support IPv4 stack for foreseeable future anyway so kinda moot point. If you game a lot p2p (i don't) you should, in theory, see lower lag.
For me personally, I work on networking startup so I'd like to be able to run IPv6 stack from my home network to test things.
If you don't currently have IPv6 service, have you looked into something like Hurricane Electric's IPv6 tunnel broker? [0] It's how I got my first IPv6 subnet, and worked really well for me. I stopped using it when Comcast finally got around to providing IPv6 for non-business accounts.
[0] https://tunnelbroker.net/
It’s been like two decades since i last used this but at the time HE 6in4 tunnel required static ipv4
It has also been about that long since I used the service. According to my very, very distant memory this requirement was implemented by having you punch in your IPv4 address into a form to configure their end of the tunnel. I know for an absolute fact that Comcast was not providing me with a static IPv4 address, and the HE.net tunnel worked just fine.
Most video games don't work with ipv6 at all, which is ironic since in theory they're the exact use case for it
Pretty much negative, I always disable it
Unfortunately Sonic does not cover the whole bay, and certainly not all SF. I am still waiting for Sonic to cover the heart of the City (Eureka Valley).
Downtown San Jose is nice - I have fibers from both AT&T and Sonic. I switched from AT&T to Sonic a couple years ago and have been impressed. I pay half what I did, get 10x the speed, and customer service is much better.
Downtown SJ has sail internet as well, great local isp!
I am on it, still a little bit pricer than Comcast, but worth every penny.
In my condo complex the cost for Sail is the same as the lowest Comcast internet plan. Possibly Sail and others have forced Comcast to go down finally, which is great.
Also the Sail support is really good, fast responses (even the CEO sometimes answers tickets).
Thanks for saying so - I got a flyer and didn't realize they are local.
I have two lit fiber cables to my house in exurban maryland and I find it hilarious that many places in the Bay Area have zero.
In most of San Francisco and parts of the East Bay, there's MonkeyBrains: https://www.monkeybrains.net/
Does Monkeybrains offer fiber now ?
I have been a customer for 14 years now. Would love to move to higher bandwidth.
It could just be mundane technical debt or just organizational bureaucracy .
I recently moved into Menlo Park and had no problems getting 2.5Gbps from ATT fiber.
I've seen a few articles about folks who started an ISP and they always talk about the physical infrastructure. But in today's world where ISP ads are touting the speeds of their wifi, it really makes me wonder what the support burden ends up being like. What's the breakdown for actual ISP issues vs issues with customer equipment?
My experience from almost a decade ago, mostly in DSL land, is that most customer calls were "my WiFi doesn't reach through the solid steel wall the router is hung against" and "how do I set up my email" and maybe "I lost the password to my WiFi again". WiFi issues were especially bad when 802.11n got finalised but there were tons of "draft n" WiFi devices out there that almost followed the WiFi spec. I still shudder when I see Atheros listed in device manager.
There were things that made the ISP I worked at special, one of them being that we pretty much defaulted to having customers hook up their own DSL, which meant spending a lot of call time helping people who have no idea what an RJ11 jack is install plugs and adapters.
I've also spent a lot of time on "the password I use for my email doesn't work on my Facebook" and "my USB printer doesn't work". People don't know who to call for tech support so they try their ISP. There was also the occasional "the internet is broken" whenever the user's home page had a different theme or design as well, those usually came in waves.
Once the modem and/or router is installed, most internet services Just Work. There are outages and bad modems and the occasional bad software update to deal with, but they're a relatively low call volume compared to what customers call about.
I quite enjoyed "I run a massive home business and require 24/7 uptime and will get extremely aggressive about this"
"Maam, if your business is that important, surely you as a responsible business owner have gone and purchased a business class internet service with 24/7 SLA. It says here, you are on our cheapest, residential VDSL service"
Which is why comcast goes to such great lengths to ensure they own as much of your network stack as they can - in my area at least, their support is capable of fully managing your router and WiFi remotely if you're leasing their equipment. I imagine this is a great boon for their ability to provide tech support (and includes a host of other "features" that don't serve direct customer needs such as a non-optional guest WiFi access point that any other comcast user can use).
This leads to fun tech support calls if you use your own equipment where you're basically proving to the support underling that you know how to run your equipment for the first 20-30 minutes before they take your issue seriously (yes, the modem light is green, yes, I've already power-cycled, yes, I'm testing on a wired connection, etc)
> proving to the support underling that you know how to run your equipment for the first 20-30 minutes
I usually speedrun this by telling them something like: I am hardwired to the modem and seeing T4s in the log.
> Great. Glad to hear you are connected via hard wire Mr. teeray.
> Please wait a moment while I check on some things on your account.
> Thank you for your patience. Can you please confirm for me that you see a green light on the top of the device? Can you tell me whether the light is blinking or is solid?
The guest wifi - Xfinity WiFi - can be disabled.
https://www.xfinity.com/support/articles/disable-xfinity-wif...
Last I checked (years ago) it turned itself back on any time the router was power cycled.
I know for a while (I switched back to consumer a few years ago) Comcast Business let you persistently opt out, but if you opted out, you couldn't use other people's APs (either "share and get access to that network" or "don't share, and don't").
Now I just use my own customer modem.
If only 'shibboleet' had caught on -.-
https://xkcd.com/806/
Back when Comcast made it absolutely mandatory to have a technician come to the house to do the install, I just chatted with the tech about computer networking and our respective home setups. This usually got me the phone number for the local tech support office along with a "Call this if the service is giving you any real issues.".
> This leads to fun tech support calls if you use your own equipment where you're basically proving to the support underling that you know how to run your equipment for the first 20-30 minutes
For analyzing support burden, I think the relevant question here is why have you even had the experience of calling tech support for a non-working connection - and that falls squarely on the non-reliability of Comcast's network.
Comcast killed my Internet during an interview video call.
Called them to ask why, and they said it was a planned outage. When was it planned, I asked? 17 minutes ago.
Exactly. That's the kind of logic that only makes sense in a metastasized corpo. The only times my non-incumbent fiber connection has gone down in 8 years have been overnight maintenance windows that only happen maybe a few times per year.
Comcast said they couldn't offer me another credit because I'd had too many outage credits in the previous month.
I don't really want any outage credits.
This is 10 years out, but I used to work on an IT help desk, that was the outsourced 24/7 helpdesk / hosting for a collection of small local/regional isps (<5000 customer rural dsl companies, local municipalities, apartments, etc) My ballpark estimate from that over 3 years working there is probably 75%+ are Not equipment related. Setting up email was a big one, people accidentally hitting the input/source button on their remote and losing their STB input setting, People needing to reboot their router, flushing DNS settings / winsock reset. These might have been the majority of cases.
other than flushing DNS / winsock resets, I don't understand how the rest of those are blockers.
I think my conception of basic tech illiteracy among the general public is vastly wrong. I generally like to believe most people are competent enough to handle these sorts of things.
Those aren't mutually exclusive things. Even if 99.9% of Comcast customers are pretty good with technology, and only 1 in 1000 customers are illiterate enough that they have trouble selecting the correct input on their TV... with 32 million customers, that means you might get tens of thousands of calls about it.
But really, internet (and digital TV) services are pervasive enough that they are no longer just for technologically inclined and resourceful people. All aspects of society are now using the internet, even the homeless, impoverished, disabled, and institutionalized.
I once spent half an hour on the phone with a dialup customer who couldn't get connected. Fortunately, they fixed their own problem - they had been entering the "letter zero" instead of the "number zero" (their words.)
Took another call from an irate dialup customer who demanded a refund - he didn't know he needed a computer to use the internet -- and had driven himself mad dialing up our modem bank with his telephone and waiting for the training tones to subside so he could begin to navigate the internet as he imagined it to work: press 1 for email, 2 for news, 3 for weather...
Despite the proliferation of smart phones and greater prevalence of home networks, I don't think the situation has changed much for a segment of the population once you get down to troubleshooting why something isn't working. The skills and the willingness to just try to fix the problem aren't there.
ISPs are weird: You don’t call the water department if your sink is backed up—you call a plumber. You also don’t call the electric company when you want to wire your finished basement—you hire an electrician. ISPs somehow became responsible for absolutely every aspect of consuming their service though. Why isn’t “home internet plumber” a thing?
Most people don't have the equivalent of home internet plumbing in general. They have a hole drilled into the wall (by the ISP) where the all-in-one modem-router-switch-wap sits on a shelf. There's probably a third party service to get ethernet run through your walls, and maybe even replace your all-in-one box with something good, but most people are just doing the equivalent of getting water straight out of the water company's tap with no plumbing.
This, and also, it's much more common for internet problems to be caused by upstream issues not in the house (partly because of the situation you describe....not much to go wrong on the users end). It's very rare that a plumbing problem is because the main water line lost pressure.
Back when I still had ISPs that provided the modem + router, every single issue I think I ever had fell into one of two categories: a modem and/or router power cycle fixed it, or it was a broader network issue that had nothing to do with me or my particular internet situation (this is omitting the most common third issue: terrible customer service problems, but that's a separate thing)
Nice analogy!
After fixing internet for some neighbors and older relatives, I've wondered if people would pay for a home network / internet handyman service. It's super frustrating, especially for older folks. They often confuse their email passwords, ISP passwords, wifi setup, etc. Also I could save them a bunch of money getting rid of services they don't use, like moving their landlines to VOIP.
VoIP doesn't necessarily require your router to be up.
For example, if you subscribe to Verizon FiOS voice, the technician will disconnect your copper phone lines and connect them to VoIP termination on your ONT.
This is true, but it's not just that. How many useless cable TV packages are people paying for, on top of Netflix, Hulu, and tons of other streaming services?
I had the same thought, and even took on a few "customers" (local folks I didn't charge, but used as a test group). If I decide to do it "for real" I will definitely need to build a relationship with a person who can run ethernet cables through walls for people. I can do that, but the time it would take would not make it worth it for me.
> After fixing internet for some neighbors and older relatives, I've wondered if people would pay for a home network / internet handyman service.
That's what I did for pocket as a kid in high school (in the mid-2000s).
I don’t understand what you mean. If you want Ethernet run through your house, or coax in more places, or access points mounted, you don’t call your ISP.
You call an electrician or a handyman or somebody and tell them you have some low voltage work.
The ISP provides a cable box and modem to most homes in the same way that the electric company sticks a meter on your wall.
> If you want Ethernet run through your house, or coax in more places, or access points mounted, you don’t call your ISP.
In the US, most do. This is a standard part of "in home installation" when first subscribing to service for all of the major providers in the US.
Example: https://forums.xfinity.com/conversations/customer-service/sc...
Having worked with the public before, I have no doubt that a lot of people likely do contact utility companies for issues inside their home. Some of them even do have repair programs with outside contractors. People often simply call whoever they have an existing business relationship with for issues related to that product/service. It may be ignorant but it isn't illogical.
Also, as the other commenter pointed out, ISPs don't terminate their service at the edge of your premises. Basically all of them today will connect one of your devices to confirm installation.
There are times when you're better off calling the local sewer department first.
In San Jose, if you see evidence that your house's main drain has backed up and you have a cleanout within 5' of the sidewalk, you're better off calling the city first before calling a plumber -- the sewer department will snake the "lateral" pipe between the cleanout and the main sewer line under the street for free.
The one time we used this the response time was very quick (in line with the 30 minute response time they cite on their website).
It's called the Geek Squad AFAIK. But most people have no understanding of how their home network works so they don't know how to decide who to call.
For the same reason you called the phone company when your phone went out, not a phone plumber.
Fibre is orders of magnitude better than DSL or cable as entire classes of problems are eliminated. Water shorting out copper pairs? Not a problem unless the water gets inside a splice and freezes causing significant bends that lower signal levels. Water getting into a cable is generally not an issue as most cables are either gell filled or have water blocking tapes. Lightning strikes are generally a non-issue since the cable isn't going to conduct a damaging charge into the ONU/ONT.
With careful selection of the customer ONU/ONT, the incidence of support calls means that it can be weeks between customer issues on smaller networks. These days my biggest support headache is in house wireless coverage. It's also the one part of internet service that most people are unwilling to invest even small amounts of money to improve. The worst are the folks that install outdoor wireless security cameras without thinking ahead to putting them on a dedicated network to avoid driving up airtime usage and congesting the main wireless AP.
Right now device manufacturers are capturing the sliver of the market that actually cares about having fast Wi-Fi. A few ISPs have partnered with these manufacturers through the time honored business model of charging $10-20/mo forever on a box that costs $200.
My electricity and water is much more reliable than my Internet service. Then again, I've never called my ISP about an issue that wasn't 100% on them, but the HN crowd is more exceptional in that sense than most people.
In my experience with small ISP's they make 99% of their own headaches, the last 1% being provided by the small consultants who work with small ISP.
For fiber customer side issues are almost all wifi related, to the point that some operators will offer in home managed wireless options.
I used to provide support in an area where a provider had purchased a VDSL network in order to convert those customers to fiber. 20% of customers remained on VDSL for various reasons. 10% of customers had been moved to a dodgy hybrid fibre/last mile ethernet solution. and the remainder were all on fibre.
70% of support issues related to the VDSL customers. 20% the ethernet customers. and the remainder of issues were almost all wifi or power related.
They had a policy of charging customers 1000 bucks or so to convert them over to Fibre. Eventually they sold the business to a larger entity. 4 weeks of VDSL complaints, and the new owners gave everyone remaining on copper a free fibre upgrade.
Actually it was only technically VDSL. What they did was drop a fibre ntd into the old vdsl node, commission each port for a different customer, and then run a Ethernet / VDSL converter over the old lead in. The "upgrade" was just using the copper as a draw wire for the fibre cable. Nothing over 100 meters.
I'd imagine it's a lot less than "Okay, let's start by going into your dialer settings..."
With fiber, the ISP can see that everything is good up to the GPON terminal. Probably the router too as most customers will just use the ISP provided one. So that leaves the ethernet interface / wifi card as the only thing that would fail and have to be ascertained over the phone, and with a local ISP its probably more cost effective to cut out all the abstractions and just have a tech stop by to check it out.
On the other side, customers have become a lot more used to self help. For example their email isn't even hosted with the ISP any more! I would think that most people would be aware that if a device works good close to the router, and not good far, the issue is wifi range. If they're still calling the ISP, you can direct them towards wifi extenders. Or if device A does not work but device B does, it's not a problem to call the ISP about. And so on.
Of course this is my idyllic view not having worked ISP tech support in a few decades...
I am always baffled by these things. Say there’s a huge company with a monopoly in your area. My first thought is “How did they get that monopoly? What happened to all the other people who must surely have had the idea to compete with them?” But no, these stories are always treating “Hey, let’s start a competing company!” like some revolutionary idea that nobody has thought of before, and that success is assured.
> What happened to all the other...
There's a huge gap between "had the idea" and "had all the technical skills, the $millions in capital, and the managerial ability to actually build it". Then there's the barrier of "and succeed". If you read between the article's lines a bit - these guys had loads of the first 3, yet they're still losing loads of money every month.
But, bigger picture, you have a good point. These articles are obviously cherry-picked stories, with an extremely optimistic "... and the little guy wins!" spin. Ars is writing for an audience of techies who are frustrated with crappy ISP's.
The capital for an ISP is surprisingly low. The main problem is getting a physical connection to your customer's house. And that's such an obvious legal minefield that no networking nerd wants to do it.
The capital can be low. Vs. the article notes that these two guys have 75 miles of fiber installed (to 1,500 potential-customer homes) and 15 local employees. Vs. currently monthly revenues of about $10K.
Yeah, obviously these guy's long prior experience - pulling fiber for other ISP's - was another critical cornerstone of their ability to go from idea to build-out.
>The capital for an ISP is surprisingly low.
Crazy generalization.
>And that's such an obvious legal minefield that no networking nerd wants to do it.
Honestly half the fun.
In the specific case of ISPs I think it's always because you won't make enough money to justify it as a big company, yet the task is too big and complicated to do it as an individual nerd.
The worst part appears to be the physical wiring. If your government has implemented loop unbundling, you're already set (probably need to do some bureaucracy and pay some affordable-at-a-stretch fees to get access to it). Otherwise, or if the loops are just crap, you have to figure out how to physically get a cable to everywhere, a task that is fundamentally laborious and legally fraught, not nerdy at all (unless lawyers are nerds) so nobody wants to do it.
Wireless ISPs are about as popular because of this. Wireless service is always worse, but you only have to install plant (physical infrastructure) at the customer's house and one central location, not all the places leading up to the customer's house. This makes it a whole lot more amenable to individual-nerd or handful-of-nerds setup.
I encourage everyone to at least think about how they would do it.
> ...you only have to install plant (physical infrastructure) at the customer's house and one central location, not all the places leading up to the customer's house.
In a rural environment, yeah, sure. Based on what I'm seeing in San Francisco, in an urban environment, you're going to be negotiating for roof space for many transceivers on many separate roofs. (I do absolutely agree that even that annoying tasks is certainly way less work than dealing with a local or state government that wants it to be impossible to run fiber through or along streets and sidewalks.)
>Wireless service is always worse
Look, wireless service is almost guaranteed to be worse, but that has more to do with dodgy operators. The technology is fantastic, and when engineered correctly largely undetectable.
That said, in my time, I can count on one hand the number of installations where I was allowed to engineer the service correctly. And I can count on all the hands in a small city the number of times I have been called to rescue something extremely stupid, like shooting a link across a construction site.
You can create a great wireless link. It still won't beat a great fiber link. I'm sure the best wireless links beat the worst fiber links, but...
>You can create a great wireless link. It still won't beat a great fiber link. I'm sure the best wireless links beat the worst fiber links, but...
When engineered correctly people tend to have absolutely no idea wireless is involved in their connection. Largely its a self inflicted branding issue. You see wireless being sold as "Fibre Extension" way too often for this reason.
Theres also factors that fibre people never consider, like mean time to restore a service. Even if you have a team of 24/7 engineers ready to mobilise, a fibre break will often take a significantly longer time to restore than a wireless outage.
Look at it this way as well: The slowest practical fiber connection is 10Gbps, individually per fiber. The fastest practical wireless connection is about 2.5Gbps, shared between all users.
Slowest practical fiber connection is 10 gig? There are still plenty of 1 gig optics in the world. Still plenty of FC8s too.
And its weird that I can remember building a 10 gig path of siklus just a few years ago. Dodgy product in terms of failover, but delivered the goods.
Telephone and cable TV companies were explicitly envisioned as regulated monopolies in most places. Then it was cheaper for them to provide Internet over their existing lines than for a new company to come in.
I didn't think I've ever seen mention of a buyout in these articles. That could be something. Franchised ISP. Maybe Comcast is incapable of servicing an area effectively, so they could say something like "we'll give you x gbps of guaranteed throughout at the datacenter (or however it works) to our main line and teach you how to setup, you cover installation and maintenance". Just because it seems like it would've been easier for these guys to do only the installation and routine maintenance. But idk I guess they don't want to because they make their money anyway
A presentation from 2020 NLNOG by Jared Mauch who did something similar in Michigan community:
* https://www.youtube.com/watch?v=ASXJgvy3mEg
2020 NANOG:
* https://www.youtube.com/watch?v=Twe6uTwOyJo
So glad to see a renewed emphasis on proper wired infrastructure. It seems the "big boys" (Verizon, T-Mobile, etc) are heavily pushing wireless and not building out new wired areas, I assume because it's less capital intensive.
Hell if there's a way to invest in Prime-One, these guys seem to have their stuff together...
> It seems the "big boys" (Verizon, T-Mobile, etc) are heavily pushing wireless and not building out new wired areas
Those are all telecom providers. It makes sense that they'd love wireless because they already have cellular infrastructure.
For anyone who doesn't know the area, Saline is adjacent to Ann Arbor, and along with Ypsilanti makes up a sort of greater Ann Arbor/UMich co-prosperity sphere. Saline is the kind of place you expect people to stand up a private fiber ISP; a place with an outer-ring suburb vibe, but far from any major metro, with lots of nerds.
If you ever have the chance to support a local ISP like this, do it! You can get some pretty sweet deals, the last time I had the opportunity to do this they threw in a /28 for "free" (agreed to two year terms)
When was the last time you had the opportunity to do this? 1998? I believe that what you're telling me is true, (and yeah, small ISPs are often really great to be customers of because of stuff like that) but -given the state of IPv4 allocations- I find it difficult to believe that it happened within the past ten, twenty years.
Less than a decade ago! The rough part is when they inevitably get bought by one of the big ISPs
This is why national ISPs like Comcast have fought/lobbied tooth and nail to prevent municipal based ISPs from being created in various states (ie, Texas). Next logical step is starting a small ISP like these people have but they have the advantage of learning the skills and process (permitting with municipality) of doing this for other ISPs. There's also the capital aspect of this, which they apparently have.
> Comcast seems to have noticed, Herman said. "They've been calling our clients nonstop to try to come back to their service, offer them discounted rates for a five-year contract and so on," he said.
go figure. their monopoly/duopoly has ended, profits dropping like a rock in area, and now they want to compete.
Only billionaires and people fooled by Peter Thiel think competition is evil.
Wow the US really has it bad when it comes to home internet. In many European countries, you can get symmetric Gbit internet for 30-40 EUR (probably less in some places), and I haven't seen a data cap in forever.
The EU is better on average, but isn't universally great either. I pay 60 EUR for 200Mbit down/20Mbit up ADSL in Amsterdam, after my 6-month discount ran out. No fiber in my neighborhood yet. There's one gigabit provider in my neighborhood (Ziggo) and they have a bad reputation. For the same price I was getting FiOS gigabit in NYC.
I have 1.1Gb/100Mb on Ziggo in Amsterdam. I've had no real issues with them, at least for me the reputation is undeserved. I pay a bit more than that, but like €20 or so. They also give me a /57 or something which is nice (if a weird allocation, but I'm not going to use that many subnets anyway.)
I have had a fibre cable poking out of the footpath in front of my apartment for a year or two now, waiting for ODF or whoever to come and install it into the building.
That's crazy, 200M asymmetric for 60 EUR is robbery.
Would it make you jealous if I tell you, that I get 10 Gbit symmetric fiber here in Switzerland (greater Zurich area) for roughly 80 USD/month with no data cap? And I can use my own router and could even go up to 25 Gbit if I want ;-) Oh, and did I mention no CGNAT and it comes with a static /48 IPv6 net?
You are talking about init7, and they did have to fight tooth and nail and take it to court to get SwissCom to provide reasonable access to the fiber.
IIRC, Init7 explained that basically the internet wants to be fast, but ISPs want to spend lots of money on special equipment to slow down your internet so that they can sell speed tiers and data caps.
The US is much more captured that Switzerland, of course. Providers in the US get billions of dollars to expand their networks and provide service, and then don't provide the service, and nothing happens.
To be honest, I'm significantly more envious of 1gig for $40 than 10gig for $80.
And my drives aren't fast enough for 25.
I'm paying under $40/mo for 10g symmetric here in southern Japan, that's including the extra charge for a public static IPv4 address (standard is a /56 IPv6 and tunneled CGNAT IPv4)
Not really, I also live in Switzerland and would have access to equivalent plans where I live :P
Depends on where in the US. Most populated places have inexpensive internet. Smaller towns have these issues because there's not much competition.
It's getting better here. Google Fiber is expanding to a lot of cities and their symmetric Gbit with no data cap is the equivalent of 60 EUR ($70).
Here symmetric 4Gbit without a data cap (NL). Best of all, you can bring your own equipment. I have my Ubiquiti Gateway Max hooked up to fiber with a media converter (yes, the Gateway Max does PPPoE etc.).
My parents live in a small, countryside village. They have fiber at the same prices (including 4Gbit symmetric, though they are happy with a cheap 200Mbit subscription).
Bay Area has sonic.net with unlimited 10Gb down & 1Gb up for only $40.
I don't think sonic has asymmetric internet anywhere. It used to be symmetric gig, now they're deploying symmetric 10gig. And the price is $49, although they just announced an increase to $59.
I had symmetric 10Gb for over a year, until they sent a new modem. When I tested the speed again, it was asymmetric. I figure it's the new modem, but the change may have been made before. I don't usually upload >1Gb, so I didn't notice when it actually happened.
*parts of the Bay Area. I'd say the majority of areas are still monopolized by Comcast, including my neighborhood of course.
If you haven't already, check to see if either Google Fiber or Monkeybrains is available in your area. Last I checked, the regs are still in place that prevent landlords from denying you access to an ISP of your choice.
Google Fiber isn't rolled out to most of the bay area. Monkeybrains is wireless with speeds significantly slower than what Comcast offers me. I've checked just about every wired ISP possible, and Comcast is the only option that services my neighborhood.
And FWIW, I own my house in the east bay — I am the landlady ;)
> Google Fiber isn't rolled out to most of the bay area.
I'm aware. And it's my understanding that -sadly- much of the Google "Fiber" deployment in the area is a WISP, just like Monkeybrains is. Quite a while back, Google Fiber bought Webpass and continued doing WISP deployment in the SFBA under the Google Fiber brand. (Because it's politically dreadfully hard to run fiber optic cables in the area.)
If you haven't contacted Monkeybrains for a minimum and expected speed quote at your site in a year or five, it's worth doing it again. It's my understanding that they aperiodically upgrade the hardware in their core network as well as the sort of hardware that they deploy at customer sites.
Monkeybrains' down-to 100/100 service is -on paper- far, far slower than the up-to 1400/40 service I was getting from Comcast, but the actual, delivered speed that I'm seeing from Monkeybrains varies between 300mbit and ~1000mbit (sustained) depending on what other folks are doing on their network. [0] I'm in a fifty-apartment building, so it's possible that they've installed faster gear on my roof than they install in smaller (or single-family) buildings. Reports on the Internet seem to be somewhat mixed, with some single-family buildings reporting ~1gbit service, and others reporting ~45mbit.
[0] Typical prime-time speed is something like 400mbit. Off-hours speed is frequently very close to 1gbit. The only time I've seen the minimum speed was when I had a poorly-crimped Ethernet cable between my router and the rest of my LAN that would intermittently only link up at 100mbit.
Seconding monkeybrains - they can usually get to houses that the other ISPs can't/won't service, and speed is pretty spiffy
This "$80 for gigabit fiber, unlimited data" is a problem for me. I want to save money, I don't need a gigabit fiber but there's no other options. If there's $30 for 300Mbps unlimited data (I'm paying $40 for optimum) option, then I'll take it.
I wish these guys the best, but I've shifted more and more to Starlink. As long as their is a clear view of the sky, it works exceptionally well and is more than enough bandwidth. Plus, I can easily take it with me anywhere I go, which includes my campervan. This is great for when you're out in the middle of nowhere, with no reception, and you need access to maps.
I wish it was a bit cheaper, but someone has to fund that trip to Mars.
I've really been looking at some of the new mobile options for when I take the family camping but I still need to get some stuff done. With that said, over the last 30 days at home we've downloaded 4.8tb and uploaded 6.2 (no torrenting). I'm sure there are thing we could do different, but two people that WFH and do some semi data involved things... really just not sure how we could make that work full time.
Getting the rights to lay the fiber seems like the most difficult part, and they don't go into much detail. Which HN user knows something about this?
I'm kind of surprised how expensive this seems. I always thought NZ ended up paying through the nose for the internet but these prices all seem roughly in line with NZ except they're in USD which is about 1:0.6 right now.
Congrats! I grew up just down the road from Saline. Exciting to see this happening on my old stomping grounds. Best of luck.
I once lived in a town with local high speed (although it was cable, not fiber). It really does make a world of difference in terms of what you pay and what kind of support you get.
It's disgusting that big telecom has been able to monopolize so much of the US for so long.
But kudos to them, that's the only way to break monopolies. Although I do wonder how much they bet on people not using all that bandwidth based on their promises.
From everything I've ever read and my (admittedly old) ISP experience, people really don't use all the bandwidth. Yes, occasionally there are exceptions.
I understand the need for independent fiber ISPs. But are gigabit speeds really necessary? For me, a 300 Mbps connection is way more than enough for a four-person family.
Every bit per second that a WAN connection is slower than a CPU interconnect is too slow and that's only half hyperbole.
But I'm a realist so I'll take what I can get.
As your remote resources get faster and faster you start using them more like local resources, which can change (often for the better) how you do things.
If all you're doing is watching netflix then 25mbps per user is probably fine.
If you're working or creating or earning, then you want your connection to be as fast as possible, your distant end to be as fast as possible, and the hardware in between to be fast as possible.
> But are gigabit speeds really necessary? For me, a 300 Mbps connection is way more than enough for a four-person family.
I'm certain that one could make a sound argument that 300 Mbps is not necessary for that four-person family, and they could make do with a much slower connection. Back in the day, folks would be asking if it's necessary for your Internet connection to be always on. After all, it's no hassle at all to plug the modem into the house phone line and unplug it when you're done!
For me, switching from a 1400/40mbps cable connection to a symmetric-but-variable 300-1000mbps Ethernet connection meant that I was doing the same sorts of things, but often spending much less time waiting for them to complete. Related to that, it also made "content creation"-esque things [0] much, much easier.
[0] Which I'm declaring is a category that includes uploading and downloading huge files while working from home as a programmer/"DevOps" guy.
Low upload speed is like a house with only one bathroom.
Gigabit is the slowest reasonable fiber speed. Even 10-gigabit is now very cheap to the point that some gigabit equipment is just 10-gigabit equipment operating at a low speed. They could artificially throttle you to even less, if it made business sense.
It's not a committed rate. Your individual line is a gigabit, but the upstream from your whole block is 10 gigabit so you can't all use it at once. Your guaranteed rate is probably have more like 20-50 Mbps, if that's what's confusing you. But it's extremely rare that everyone tries to use their gigabit all at once.
If it's a Passive Optical Network, you might be sharing a gigabit download with your block - you all share the same fiber - and you get substantially less than a gigabit upload due to the need for timeslotting. Gigabit PON is obsolete though, now you'd get at least 10G PON.
It's usually the upload speeds that are the bottleneck. When the upload is only 10 Mbps any video upload or large file upload will saturate the connection. In addition to having to wait for the video to finish uploading, if you don't manually rate limit the upload job speed or have anti-bufferbloat measures deployed any concurrent voice or video calls will suffer. During Covid shutdowns four people video chatting at once was no longer far fetched.
The scarcity of upload speed is why I learned how to setup OpenWrt's SQM feature, and upgraded to the highest package my ISPs offers.
Interesting. Yes, I guess the upload speed definitely matters, which cable/DSL don't provide.
I myself have fiber, and speeds are much less than 500 Mbps, but symmetric (with likely maximum two people using video calling at a time), so haven't faced that issue.
That's a bit more than I have (Starlink), but anytime a kid is downloading a big Steam game or big ISO file or something, everything else slows to a crawl. I also occassionally rsync large directories to/from cloud storage and that can also saturate. I've tried setting rules/priorities but it's a constant game of whack-a-mole
For convenience, the rclone tool is nice for most cloud storage like google and stuff that make rsync annoying[0]
rsync also offers compression[1], and you might want to balance it depending if you want to be CPU bound or IO bound. You can pick the compression and level, with more options than just the `-z` flag. You can also increase speed by not doing the checksum, or by running without checksum and then running again later with. Or some intervaling like daily backups without and monthly you do checksums.
If you tar your files up first I have a function that is essentially `tar cf - "${@:2}" | xz -9 --threads $NTHREADS --verbose > "${1}"` which uses the maximum `xz` compression level. I like to heavily compress things upstream because it also makes downloads faster and decompression is much easier than compression. I usually prefer being compute bound.
Also, a systemd job is always nice and offers more flexibility than cron. It's what's helped me most with the wack-a-mole game. I like to do on calendar events (e.g. Daily, Weekly) and add a random delay. It's also nice that if the event was missed because the machine was off it'll run the job once the machine is back on (I usually make it wait at least 15 minutes after machine comes online).
[0] https://rclone.org/
[1] https://unix.stackexchange.com/a/292020
Thank you! I do love rclone a great deal, and need to start using it more. I mostly use rsync right now because I've got some Contabo storage instances and ssh is already set up, so it's nice and easy, and I've already got the rsync commands burned into memory from decades of use :-) I also typically rsync with the same command to an external hard drive for an on-site backup.
Great tips! I'll definitely be using your tar command
Oh, that reminds me. When I had an Android (I already regret my iPhone) I used termux to write a very basic script to rsync data to machines and do so based on if connected to WiFi (with whitelisted SSIDs). Pretty easy to write and then schedule a cron job. Can be nice to put that onto other peoples devices and give them automated backups. Makes it FAR easier to do the 321 backup strategy. Also pretty easy to build a tracking app for a lost phone that way (like if haven't connected to home WiFi in x days send you an email). Both work really well when also using tailscale.
The article says they're a 10 person family.
Depends on what you're doing, right?
Let's take video streaming. I have a pretty compressed version of Arrival that's at 2GB and is a 4k movie ~2hrs long (the original file was ~2x the size). To stream that we need to do 2000Mb / (3600s * 2) = 277.8Mb/s. This also doesn't account for any buffering. This is one of my smaller 4k videos and more typical is going to be 3Gb-5Gb (e.g. Oppenheimer vs Children of Men). Arrival is pretty dark and a slow movie so great for compression.
Now, there's probably some trickery going on that can get better savings and you'll see used with things like degrading the quality. You could probably drop this down to 1.5Gb and have no major visual hits or you can do a variable streaming and drop this even more. On many screens you might not notice a huge difference between 1440 and 4k, and depending on the video, maybe even 1080p and 4k[0].
For comparison, I loaded up a 4k YouTube video (which uses vp9 encoding) and monitored the bandwidth. It is very spiky, but frequently jumped between 150kbps and 200Mbps. You could probably do 2 people on this. I think it'd get bogged down with 4 people. And remember, this is all highly variable. Games, downloads, and many other things can greatly impact all this. It also highly depends on the stability of your network connection. You're paying for *UP TO* 300Mbps, not a fixed rate of 300Mbps. Most people want a bit of headroom.
[0] Any person will 100% be able to differentiate 1080p and 4k when head to head, but in the wild? We're just too used to spotty connections and variable resolutions. It also depends on the screen you're viewing from, most importantly the screen size (e.g. phone).
Blu-rays don't do anywhere near 277Mb/s.
First, if it was 2GB * 2 for the source of your recompressed copy, that's 4GB * 8 bits per byte = 32 Gigabits (Gb), or 32,000Mb. Two hours in seconds is 60 * 60 * 2 = 7,200 seconds.
32,000 / 7,200 is 4.444Mb/s. Streaming your 2 hour long 4GB movie could be done with ~5Mbit. A 1Gb/s connection could handle streaming ~200 of these movies.
Going back to Blu-rays as a source, an Ultra HD Blu-ray maxes out at 144Mbit but in reality most movies are encoded at a much lower bitrate. Most movies will cap out around 40-50Mbit. You could do 20 of these straight Blu-ray movies on a 1Gb connection.
https://en.wikipedia.org/wiki/Ultra_HD_Blu-ray#Specification...
> I have a pretty compressed version of Arrival that's at 2GB and is a 4k movie ~2hrs long (the original file was ~2x the size). To stream that we need to do 2000Mb / (3600s * 2) = 277.8Mb/s.
Your math is WAY off.
2 gigabytes / 2 hours is only about 2.22 megabits/sec.
> 2000Mb / (3600s * 2) = 277.8Mb/s
2000Mb / 7200s = 0.278 Mb/s, or 277.8Kb/s
> It is very spiky, but frequently jumped between 150kbps and 200Mbps. […] I think it’d get bogged down with 4 people
That’s just burst buffering as fast as it can, you didn’t capture the average. It doesn’t suggest it would slow down with 4 people. 4K on YouTube takes 20Mbps, so to parent’s point, you’ll have plenty of bandwidth to spare if 4 people do this at the same time on a 300Mbps line.
I mean the question is why limit it? I'm paying under $40/mo for 10 gigabit, bandwidth shouldn't be expensive, the expensive part is digging the last mile.
> Any time any customers are experiencing an outage, maybe because of a lawnmower that cut their service line
Whoa, they are laying cable just like that, in the open?
Ha ha, I misread it as "Two guys hated using CompuServe, so they built their own ISP". Wrong millennia ...
I'm a simple man. I see Comcast hate, I upvote
I hate Comcast, I need to do this too
I have to admit, everytime I hear the "Two guys hated x, so they built their own!" I see the XCKD cartoon https://xkcd.com/927/
It's not a fair comparison; competition can drive price down, but I pessimistically just see two guys who'll inevitably join the Comcast billionaires club. That's just where these "small guys" end up.