I have worked with several development teams over the last few years in creating real estate products for our clients. When clients required MLS data, RETS was the method used to obtain the information. While requesting access to the data from the MLS is always challenging, it is a fairly common task on the project list.
Recently, we partnered with a client who believed in using modern technology. Cool! Since the WAV Group and I are proponents of RESO, let’s use the RESO Web API.
In practice, this decision became an experience that confirms the RESO Broker Advisory Group’s new initiative is desperately needed.
Off We Go
Our clients were in rapid prototype mode to launch a minimum viable product (MVP) for a proof of concept (POC). Their idea of producing a great real estate app still needed to be validated against their business model, and had to answer the age-old question, will people use the product and service.
Since our client’s POC evaluations required data from a single MLS, the decision was to obtain the data from the MLS directly. We solicited the appropriate data agreement and had the sponsoring broker sign-off on the agreement.
Simple enough, right? It started that way. Obtaining the keys to the data went smoothly. The dev teams had access to RESO’s and the data provider’s API documentation. They were having success with querying the MLS data in real-time for a mobile native or web applications.
Not a problem, right? Working with web APIs today is a reasonably ubiquitous development practice…except when using replication with MLSs and MLS data.
Development Team Trials with Web API
When we arrived at the timeline of the project to ingest MLS data through replication, all hell broke loose with the dev team and the MLS.
LOL! All hell didn’t break loose, but a series of what I thought was going to be straightforward tasks became lost time, additional frustration, and extra effort to resolve.
Rate Limit Trial
The dev team surfaced with a challenge of how to access all the listings. The snag was the query would fail based on the allowed rate limits set on the data providers API end-point. Solving for this little barrier was dissolved with an explanation of the nextLink URL found in the JSON header. The data provider exposes the nextLink URL to manifest the next series of records. Problem solved!
Grasping Too Much Data
Another predicament arose when the dev team began to use the $expand System Query Option to retrieve listing and agent data with a single query. You can use $expand in the URL on a Web API end-point, but specific data providers block usage of this option when using an end-point dedicated for replication. The obstacle presented by the dev team was the data was incomplete when using the replication Web API end-point.
Thinking there was a problem with either the data provider or the level of access provided by the MLS, I revived up Postman to retrieve responses from the URL query for replication and non-replication containing a $expand option.
With a little bit of Python and Pandas library, I extrapolated the JSON response and created a CSV file from each URL. All the columns matched perfectly for the listing data.
What the dev team observed and questioned was the agent information did not exist in the replication query response. This result is correct, and the data provider’s documentation designs this result. Obtaining Agent information commands a unique replication query against the Members resource.
MLS Challenges with Web API
The sticky point with the MLS occurred because of a lack of communication. MLSs seem to prefer contact through email, which I loath. Email is synchronous and doesn’t allow for a simultaneous two-way conversation. Unfortunately, the MLSs seem to have gravitated towards email to interact with customers. I never had this challenge with Stellar MLS when I was a CTO for a brokerage. Kudos to the Stellar MLS team! I appreciate you guys!
Challenge #1
Difficulty emerged when the dev team tried to use the replication Web API end-point. When they sent queries to request data, they would receive an “Invalid permissions” response.
The issue? It seems some MLSs now require everyone to obtain a special ‘Approval’ to use the replication API end-point. I am not sure why there is a need for that type of approval. Why couldn’t a request for replication be identified and approved during the data agreement?
When questioning the errors, the MLS required a use case to justify access to the data using replication. I wrote a smart little use case, and the project was on track again, or so I thought.
Challenge #2
As the dev team moved forward with replication, we noticed there were columns without listing information. When the MLS provided the keys, the type of data feed approved was only for IDX. IDX only contains a small subset of all the MLS information necessary for brokerages and agents to market property on the Internet. IDX doesn’t include data like OccupantType or OccupantName. These were two fields required by the client’s use case.
The agreement did have a section to indicate if the data would be used for a Public facing web site. The signed data agreement did not have this option selected.
Another section of the data agreement had a part to allow comments. This section of the signed agreement stated the information was “for the agent’s use only.”
Another set of emails resulted in the MLS asking for another use case. Once we submitted another crafty use case, we finally had the information needed for the development team.
A plight of delays, misunderstandings, and the need for approval to replicate with the Web API will someday be seamless and painless.
Summary
I wish there were concrete steps to follow for working with MLSs. In my experience, each MLS is uniquely different. Data agreements, handling support requests, and how and what they communicate all vary greatly.
I do suggest following these guidelines:
- Before asking for data, build a detail use case to explain “what, who, where, why, and how” the MLS information creates a viable industry product. If you are a technology company, do this before contacting the MLS or approaching a broker.
- Be very clear in the use case of what type of data feed is necessary. There are four types, IDX, VOW, Broker Back-office, and Broker Only.
- Make sure your development team understands the RESO Web API, RESO Data Dictionary, the data providers API documentation, and the difference in replication and real-time Web API URLs. Review your database models early in the process and don’t bypass this step.
- If your use case requires the replication of data via Web API, check if approvals are needed when submitting the use case.
Last point that I want to make. Join RESO and become a member of the Broker Advisory Workgroup. This workgroup is structuring an education curriculum on all these points. If the program “Working with Real Estate Data” was in place, all the above pitfalls would have never existed.
Instead of driving yourself to drink, you can call us at the WAV Group. Our experience with MLS data can assist in streamlining the MLS data access process for you. Contact Victor, Marilyn, Kevin, or David for a consultation.
