> Doesn’t SpaceX and other satellite broadband kind of make this map irrelevant?
(Not the person you replied to)
No, it doesn't. Fundamentally, no matter how many satellites SpaceX shoots up, it's still a geographically shared medium with significantly more latency and lower bandwidth than DSL or direct fiber.
It's a stopgap for extremely rural areas that even in a thousand years couldn't recoup the investment of laying fiber, RVs, EMS, the military, ships, planes... but that's it. It can't ever beat a good 1 GBit/s fiber, much less the 10 Gbit/s you can get in some particularly well developed places.
When I checked last year the per-address geodata (sans addresses, sadly. Those are only available by license from CostQuest) was available via the map API. Only in small amounts (high zoom levels) however, and rate-limited. If someone wanted to perform a public service at the risk of pissing off the powers that be (see Aaron Swartz) it could be scraped.
Because transparency and privacy are in direct conflict, and everyone has a "well ackshually" on where you should fall on that spectrum.
If you require ISPs to hand over customer and connection performance data to the government you have government spying. If you have a third party estimate the data and sell it to the government, you have a proprietary dataset with no insight into its accuracy. If you let ISPs self report aggregate data they lie to make themselves look better while also convincing their competitors to build out in unprofitable markets. If you give nerds bulk access to data they will figure out a way to abuse it and pitch a YC startup to figure out when you are watching porn and better target ads. Everyone loses in this game.
The real solution is you shouldn't care about broadband mapping. Change government broadband incentives to a post-paid reimbursement for delivering fiber to previously unserved homes. Make net neutrality a constitutional amendment. Make FCC spectrum auctions based on who will bid to provide the most free service to underserved users.
> If you require ISPs to hand over customer and connection performance data to the government you have government spying.
What customer information? "We have a customer at this address and the min/avg/max download speed their connection has had this year was X/Y/Z" doesn't strike me as particularly sensitive information, nor PII.
> "We have a customer at this address and the min/avg/max download speed their connection has had this year was X/Y/Z"
Not even that. The data could be "With no more than a 1-hour router installation on premise, we (ISP Alpha) would be capable of serving this address at 500 Mbps download speed with 99% uptime, 800 Mbps down with 95% uptime, and 1 GB Mbps with 80% uptime." Ditto for upload speed. That's not customer information. Whether anyone at "this address" actually uses ISP Alpha internet right now doesn't matter until the customer disputes the advertised speed.
Given all the doom predictions that failed to pass back in 2017, do we really need to add it to the constitution?
>Make FCC spectrum auctions based on who will bid to provide the most free service to underserved users.
Sounds this is just a redistribution program with extra steps. The cost to provide such free service has to be paid by someone, and it's going hit the whoever is just above the "underserved" threshold (ie. the middle class) the hardest.
> The real solution is you shouldn't care about broadband mapping. Change government broadband incentives to a post-paid reimbursement for delivering fiber to previously unserved homes.
You can't tell which homes are currently unserved without collecting something.
> If you require ISPs to hand over customer and connection performance data to the government you have government spying.
Mapping speeds requires knowing what speeds the ISP is capable of providing with XY% uptime. Customer performance data doesn't have to enter the picture, unless the customer decides to file a dispute about false advertisements about bandwidth, for which case mapping laws should grant customers a private right of action as well as standing for class actions.
> If you require ISPs to hand over customer and connection performance data to the government you have government spying
What? No. At least in the case of DSL, the result of link training data is enough - it doesn't reveal anything about the actual person who has ordered a specific broadcast contract, all that the government gets to know is "phone connection X in building Y has usable upstream bandwith A and downstream bandwidth B".
Exactly, I don't know why you are being downvoted. There is data that has better to be secret, even though it is funded by public money. No controversy in that.
While they raise god points, the data itself is probably less accurate than the dataset owned by Ookla/speedtest.
The addresses and map really are pretty irrelevant. You can get all that data from open street maps. The real nice to have is service location and speed, which can be retrieved from Ookla and probably google or the data brokers.
Some data providers can map IPs down to addresses, and they probably have a good idea of how fast the actual connection is as well.
Why are we paying some lame company too much money to do this? That's really the question that should be asked.
Why is this complicated? Public bodies fund all kinds of things under terms that don't create public domain IP rights to the work product data. When the federal government buys a copy of Time Magazine, it can't just publish it. What's public depends on the terms of the deal.
I don't know whether the deal FCC and NTIA cut here is good --- is 93MM a lot for the data that got generated? But that's the question you'd have to answer before reaching a conclusion on whether the underlying data, which has value outside this project, should reasonably have been priced into it, or whether the USG got a price break for letting Costquest keep it.
It boils down to the fact that the United States does not have (public domain) knowledge of every address in the country.
The USPS knows about deliverable addresses but won't give that information to the federal government because then it'd be public domain and they would lose several of their primary data moats (Zipcodes, addresses, delivery routes, for example). The Census has very complete knowledge of every address, but won't give it up because it's illegal (see Title 13 of the US Code). There is an ongoing attempt by the DOT to collect a National Address Database (https://www.transportation.gov/gis/national-address-database) by collecting information from the address assigning authorities (usually county governments), but it's incomplete and unlikely to ever be complete because of holdout/underfunded local governments.
There are several address datasets that are private (Google has a fairly complete one, FedEx/UPS probably have the most complete, TomTom, CostQuest, etc.). I started https://openaddresses.io/ to try and collect them (NAD is based off this idea) into an open-licensed dataset.
The broadband companies have records that say "this address is connected to this network, which could theoretically have this service level", but (a) they won't/can't tell you where they think the address is and (b) won't spend the time to match their address string format with the government's address because both are private data.
Finally, without the address -> location data, even if we could get broadband providers to tell us what service is available at each address, we couldn't put that service level on a map because we don't know where the address is.
----
The Markup published some work in 2022 where they used OpenAddresses to use ISP's own tools to gather per-address service offerings and put them on a map. This is what the FCC's broadband map should be doing, but can't for the above (and political) reasons: https://themarkup.org/show-your-work/2022/10/19/how-we-uncov...
> The USPS knows about deliverable addresses but won't give that information to the federal government because then it'd be public domain and they would lose several of their primary data moats
Why does the USPS need a moat? It's not a private, profit-seeking enterprise
A major political party and several large corporations are opposed to increasing federal funds to the post office. Unless you can spin it as national defense, notable federal funding for any group is unlikely.
"Education is key to every aspect of national defense, and broadband maps are a critical element of seeing that our future soldiers, laborers, and engineers have the resources they need in order to be the best and brightest -- on the battlefield, in the factory, and in the office designing the next generation of war machine."
The other major political party likes its make work program for its voter base, so it’s unlikely to accept a one time payment vs guarantees of organizational stability.
You...are aware that the US military is a rural welfare program, right? It's designed to suck in skilless, poorly educated rural (and to a lesser degree, urban) young men...and to employ adults making munitions, gear, vehicles, and weapons systems.
For example, just about the only reason Susan Collins is still a congressional representative is because she reliably keeps defense spending rolling into the Portsmouth naval shipyard which in turn is fed by a large network of suppliers.
I like the part where we're expected to just believe these agencies/departments aren't sharing and circulating address data internally, simply because they say they aren't. As if they've never misled and outright lied to the public thousands of documented times in the past.
I don’t know if the following is related, but I find it very strange that title insurance is necessary. I would assume that both municipal and state governments would know exactly who owns what piece of property/natural resource rights/air rights/other property-related right at the click of a button.
> I don’t know if the following is related, but I find it very strange that title insurance is necessary. I would assume that both municipal and state governments would know exactly who owns what piece of property/natural resource rights/air rights/other property-related right at the click of a button.
They only need to know well enough to send a tax bill. If someone is paying property taxes, that's good enough for the municipality. It's not good enough when people start suing each other, at which point the municipality very much does not want to own the liability for any mistakes.
Consider the simple situation in which Alice sells property to Bob, without revealing (intentionally or mistakenly) that Carol has a lien against the property. If Carol comes to collect and Alice cannot pay, then Carol now also has a claim to the property.
Things get even more complicated with fraud. Lets say that Eve manages to fraudulently record a transfer from Alice to Eve, and then sells the property to Bob. Bob has not committed fraud, but Alice also should own it. In systems in which the government registry is indefeasible, then Bob owns it; Alice may recover money (see above why municipalities don't want to be liable), but not the property (had Eve been caught before selling the property then the fraud would have invalidated her claim, but Bob hasn't committed any fraud).
Of course they know. They also wisely refuse to accept responsibility for the reprecussions of being wrong.
This is the whole point of title insurance. You aren't paying for the answer. You're paying somebody to accept massive liability if the answer is wrong.
There is something called the "Torrens Title System" where the government does accept some part of this liability, subject to a huge raft of weird terms and conditions that the legislature can change unilaterally after the fact. It has only been successful in countries lacking constitutional limits on eminent domain, since in those places landowners are already subject to legislative whims anyways:
It has been a flop in every US jurisdiction that has tried it.
When given a choice between transferring responsibility to somebody who can be sued (states can't) and gives you a contract, vs someone who can't and won't, it turns out people prefer the former.
> The Markup published some work in 2022 where they used OpenAddresses to use ISP's own tools to gather per-address service offerings and put them on a map. This is what the FCC's broadband map should be doing, but can't for the above (and political) reasons
I don't know if it's still the case, but ISP's tools for address offerings were unreliable enough that realtors have suggested getting an install estimate in writing if the seller didn't have broadband.
No major ISP will give you an install commitment in writing.
I wouldn't buy a house based on a mere estimate of the availability of something this critical.
It's a gigantic clusterfuck.
The realtors know they can't fix this, but they can't exactly let it impede the source of their paycheck either. I absolutely would not take advice of any kind on internet access from a realtor.
When we did "Remote Encoding" for the USPS one of the early common "modes" was asking for Zipcode + 4 + Last 2 of House Number. The administrators explained that this 11 digit number was enough to uniquely identify most deliverable addresses "boxes" in the United States.
> It boils down to the fact that the United States does not have (public domain) knowledge of every address in the country.
i suspect licensing problems but this sounds too absurd to be true. as you said, there are multiple companies that have said dataset, at least one is under goverment control, saying nothing of the biggest defense/intelligence complex in the world which probably has coordinates of every address on the planet.
I like BAG. It is a shame that our WOZ-waarde (municipal home valuation) isn't a similar centralised system but that every municipality does their own thing. Similar for zoning, charge points and traffic signage / traffic data.
I don't have such a conspiratorial take on this. Based on my previous experience in GovTech, I think this is what happened:
The project team specs out a procurement. The procurement team gets bids and handles the bureaucracy of awarding the contract. After reviewing bids, the project team selects or shortlists finalists. A special part of the procurement team then handles negotiations. The negotiators usually know little to nothing about the project other than costs. In order to get a lower price, they will negotiate away things that the project team considers important but aren't spelled out in the requirements.
Some of those things that gets easily negotiated away are rights for designs and data, even when gov't pays for their development. Negotiators are just looking for the lowest price for this particular project at this time. After all, there may or may not be extra costs in the future; they can claim wins on lower definite costs right now. Bidders know this game, and are ready and waiting to charge extra bucks for change orders down the line.
I was a lead on a multi-million dollar open-first project for gov't. I was shocked how easily someone I'd never met gave away the rights to reproduce mounting brackets. And it was no surprise that we were locked into buying extra parts that could have been fabbed elsewhere for cheaper.
I'm confused. The articles talks about the data not being available for download and cites another article[0] which talks about it being available only for a fee.
However if you go to the maps you see the data available for download and no fee[1].
Cool project! Looking at starting a group of people to lobby my small city to invest in muni fiber. Would love to see what you have if it ends up on GitHub.
Quick typo fix on the main page: neigbors is missing an “h” (neighbors).
49 comments
[ 4.0 ms ] story [ 102 ms ] thread(Not the person you replied to)
No, it doesn't. Fundamentally, no matter how many satellites SpaceX shoots up, it's still a geographically shared medium with significantly more latency and lower bandwidth than DSL or direct fiber.
It's a stopgap for extremely rural areas that even in a thousand years couldn't recoup the investment of laying fiber, RVs, EMS, the military, ships, planes... but that's it. It can't ever beat a good 1 GBit/s fiber, much less the 10 Gbit/s you can get in some particularly well developed places.
If you require ISPs to hand over customer and connection performance data to the government you have government spying. If you have a third party estimate the data and sell it to the government, you have a proprietary dataset with no insight into its accuracy. If you let ISPs self report aggregate data they lie to make themselves look better while also convincing their competitors to build out in unprofitable markets. If you give nerds bulk access to data they will figure out a way to abuse it and pitch a YC startup to figure out when you are watching porn and better target ads. Everyone loses in this game.
The real solution is you shouldn't care about broadband mapping. Change government broadband incentives to a post-paid reimbursement for delivering fiber to previously unserved homes. Make net neutrality a constitutional amendment. Make FCC spectrum auctions based on who will bid to provide the most free service to underserved users.
What customer information? "We have a customer at this address and the min/avg/max download speed their connection has had this year was X/Y/Z" doesn't strike me as particularly sensitive information, nor PII.
Not even that. The data could be "With no more than a 1-hour router installation on premise, we (ISP Alpha) would be capable of serving this address at 500 Mbps download speed with 99% uptime, 800 Mbps down with 95% uptime, and 1 GB Mbps with 80% uptime." Ditto for upload speed. That's not customer information. Whether anyone at "this address" actually uses ISP Alpha internet right now doesn't matter until the customer disputes the advertised speed.
Given all the doom predictions that failed to pass back in 2017, do we really need to add it to the constitution?
>Make FCC spectrum auctions based on who will bid to provide the most free service to underserved users.
Sounds this is just a redistribution program with extra steps. The cost to provide such free service has to be paid by someone, and it's going hit the whoever is just above the "underserved" threshold (ie. the middle class) the hardest.
You can't tell which homes are currently unserved without collecting something.
> If you require ISPs to hand over customer and connection performance data to the government you have government spying.
Mapping speeds requires knowing what speeds the ISP is capable of providing with XY% uptime. Customer performance data doesn't have to enter the picture, unless the customer decides to file a dispute about false advertisements about bandwidth, for which case mapping laws should grant customers a private right of action as well as standing for class actions.
That's already the case though...
What? No. At least in the case of DSL, the result of link training data is enough - it doesn't reveal anything about the actual person who has ordered a specific broadcast contract, all that the government gets to know is "phone connection X in building Y has usable upstream bandwith A and downstream bandwidth B".
The addresses and map really are pretty irrelevant. You can get all that data from open street maps. The real nice to have is service location and speed, which can be retrieved from Ookla and probably google or the data brokers.
Some data providers can map IPs down to addresses, and they probably have a good idea of how fast the actual connection is as well.
Why are we paying some lame company too much money to do this? That's really the question that should be asked.
I don't know whether the deal FCC and NTIA cut here is good --- is 93MM a lot for the data that got generated? But that's the question you'd have to answer before reaching a conclusion on whether the underlying data, which has value outside this project, should reasonably have been priced into it, or whether the USG got a price break for letting Costquest keep it.
The USPS knows about deliverable addresses but won't give that information to the federal government because then it'd be public domain and they would lose several of their primary data moats (Zipcodes, addresses, delivery routes, for example). The Census has very complete knowledge of every address, but won't give it up because it's illegal (see Title 13 of the US Code). There is an ongoing attempt by the DOT to collect a National Address Database (https://www.transportation.gov/gis/national-address-database) by collecting information from the address assigning authorities (usually county governments), but it's incomplete and unlikely to ever be complete because of holdout/underfunded local governments.
There are several address datasets that are private (Google has a fairly complete one, FedEx/UPS probably have the most complete, TomTom, CostQuest, etc.). I started https://openaddresses.io/ to try and collect them (NAD is based off this idea) into an open-licensed dataset.
The broadband companies have records that say "this address is connected to this network, which could theoretically have this service level", but (a) they won't/can't tell you where they think the address is and (b) won't spend the time to match their address string format with the government's address because both are private data.
Finally, without the address -> location data, even if we could get broadband providers to tell us what service is available at each address, we couldn't put that service level on a map because we don't know where the address is.
----
The Markup published some work in 2022 where they used OpenAddresses to use ISP's own tools to gather per-address service offerings and put them on a map. This is what the FCC's broadband map should be doing, but can't for the above (and political) reasons: https://themarkup.org/show-your-work/2022/10/19/how-we-uncov...
Why does the USPS need a moat? It's not a private, profit-seeking enterprise
https://www.uspsoig.gov/focus-areas/did-you-know/postal-serv...
For example, just about the only reason Susan Collins is still a congressional representative is because she reliably keeps defense spending rolling into the Portsmouth naval shipyard which in turn is fed by a large network of suppliers.
They only need to know well enough to send a tax bill. If someone is paying property taxes, that's good enough for the municipality. It's not good enough when people start suing each other, at which point the municipality very much does not want to own the liability for any mistakes.
Consider the simple situation in which Alice sells property to Bob, without revealing (intentionally or mistakenly) that Carol has a lien against the property. If Carol comes to collect and Alice cannot pay, then Carol now also has a claim to the property.
Things get even more complicated with fraud. Lets say that Eve manages to fraudulently record a transfer from Alice to Eve, and then sells the property to Bob. Bob has not committed fraud, but Alice also should own it. In systems in which the government registry is indefeasible, then Bob owns it; Alice may recover money (see above why municipalities don't want to be liable), but not the property (had Eve been caught before selling the property then the fraud would have invalidated her claim, but Bob hasn't committed any fraud).
This is the whole point of title insurance. You aren't paying for the answer. You're paying somebody to accept massive liability if the answer is wrong.
There is something called the "Torrens Title System" where the government does accept some part of this liability, subject to a huge raft of weird terms and conditions that the legislature can change unilaterally after the fact. It has only been successful in countries lacking constitutional limits on eminent domain, since in those places landowners are already subject to legislative whims anyways:
https://en.wikipedia.org/wiki/Torrens_title
It has been a flop in every US jurisdiction that has tried it.
When given a choice between transferring responsibility to somebody who can be sued (states can't) and gives you a contract, vs someone who can't and won't, it turns out people prefer the former.
I don't know if it's still the case, but ISP's tools for address offerings were unreliable enough that realtors have suggested getting an install estimate in writing if the seller didn't have broadband.
I wouldn't buy a house based on a mere estimate of the availability of something this critical.
It's a gigantic clusterfuck.
The realtors know they can't fix this, but they can't exactly let it impede the source of their paycheck either. I absolutely would not take advice of any kind on internet access from a realtor.
i suspect licensing problems but this sounds too absurd to be true. as you said, there are multiple companies that have said dataset, at least one is under goverment control, saying nothing of the biggest defense/intelligence complex in the world which probably has coordinates of every address on the planet.
Muniplicities are the data owners, but data is also collated nationally.
See [0] for a viewer of this public data.
Also, all our surveying/road data is managed in a similar fashion, and this is regularly imported in e.g. openstreetmap.
[0] https://bagviewer.kadaster.nl/lvbag/bag-viewer/?zoomlevel=1
The project team specs out a procurement. The procurement team gets bids and handles the bureaucracy of awarding the contract. After reviewing bids, the project team selects or shortlists finalists. A special part of the procurement team then handles negotiations. The negotiators usually know little to nothing about the project other than costs. In order to get a lower price, they will negotiate away things that the project team considers important but aren't spelled out in the requirements.
Some of those things that gets easily negotiated away are rights for designs and data, even when gov't pays for their development. Negotiators are just looking for the lowest price for this particular project at this time. After all, there may or may not be extra costs in the future; they can claim wins on lower definite costs right now. Bidders know this game, and are ready and waiting to charge extra bucks for change orders down the line.
I was a lead on a multi-million dollar open-first project for gov't. I was shocked how easily someone I'd never met gave away the rights to reproduce mounting brackets. And it was no surprise that we were locked into buying extra parts that could have been fabbed elsewhere for cheaper.
However if you go to the maps you see the data available for download and no fee[1].
What am I missing?
[0] https://potsandpansbyccg.com/2024/02/14/shouldnt-broadband-m... [1] https://broadbandmap.fcc.gov/data-download/nationwide-data?v...
Sharing pre launch idea for comment, https://fiber2.me
Open for questions / feedback
That’s my biggest struggle atm, both for technical and non technical folks.
Would putting the code on GitHub help here ?
Still can’t verify what’s deployed on the backend…
Quick typo fix on the main page: neigbors is missing an “h” (neighbors).