how to call fabric rest apis from python notebooks.
introduction.
Now that Microsoft pushed Fabric REST APIs to public preview, I thought I’d provide a simple Python script on how to call them. It really is no rocket science, while it’s worth noticing that some of the API calls do not (yet?) work with Service Principals. In fact, the Fabric API groups Core and Admin are the ones that seem to currently work when authenticating with Service Principals. My guess is that in the future, Microsoft will add the Service Principals capability for the other Fabric API group Lakehouse as well. For the authentication part, especially on how to get the access token from Entra ID (former Azure Active Directory) for a Service Principal, I highly recommend checking out the series of articles by datalineo. Lastly, as already mentioned the API is still in public preview, at the time of writing this article.
prerequisites.
1. A Fabric capacity and workspace
2. A JupyterLab Notebook (of course, you can also use Fabric Notebooks)
3. Optional: A Service Principal that has appropriate access rights. You need this one if you do not want to use a user’s credentials when fetching a token (check out the very end of the Showtime part, if you are interested in a solution without a Service Principal). If you wanna use a Service Principal, take a look at the appendix on how to set this up.
plan of action.
1. What’s the goal?
The goal is to call Fabric REST APIs from a Python notebook. As seen in the example below, we called the Get Workspace API returning Fabric workspaces with the help of a Service Principal. Note, if you fancy a solution without a Service Principal, we do have an approach for that in this article, too.
2. The script
Below, the script with the Service Principal authentication. Note, the tenant_id, client_id and client_secret you can retrieve from the Service Principal page in Azure. Check out the appendix on where exactly you can find them. Also, in the script below, we call the specific Get Workspace API at the very bottom of the code snippet (line 13). If you’d like to call another API, you need adapt that line.
If you are using a Fabric Notebook you can choose to fetch a token without a Service Principal as well. You can do this by utilising the mssparkutils:
Another way is to use the FabricRestClient() via semantic-link as described by Sandeep.
3. Showtime
Here, we executed the Python script with the help of a Service Principal, normalized the json response and printed the dataframe:
Here another one, where we call the Get Items API. This call lists all the different items that are present in a certain workspace, like lakehouses, semantic models, pipelines, data flows, etc.
If you enable the Allow service principals to use read-only admin APIs in the Admin settings, you can also run APIs from the Fabric API Group Admin.
Here, we list all the items with the Get Items API from the Admin Fabric API Group:
However, if we take a call from the Fabric API group Lakehouse, we end up with an error saying the operation is not supported by the principal type. Below, we tried to list all the tables of a certain lakehouse:
We already explained this (current?) limitation: It appears only the Core and Admin Fabric API Groups work with Service Principals at time of writing this blog post. Now for this exact call (listing all the tables in the lakehouse), you could also query the tables from the connected semantic model via semantic link’s SemPy. After all, it exposes similar information. I tried to use SemPy from a Jupyter notebook (meaning outside of Fabric), but I did not manage to run it successfully due to authentication issues. So in contrary to the scripts above, the one below is not run in a Jupyter Notebook, but instead directly in Fabric:
Generally, I’d recommend using SemPy whenever possible, since you skip the authentication bit. Obviously, SemPy also needs to do some authentication, but it does it under the hood.
Talking about authentication, in a Fabric Notebook, you could also utilise mssparkutils to get an access token. Then, it is the current user that fetches the token. That way you should also be able to query the APIs from the Lakehouse Fabric API group. The interesting thing here is that if you run the notebook from a Fabric Data Pipeline, it actually succeeds, too! So, this approach could indeed be a viable alternative to directly calling the Fabric REST APIs from a Fabric Data Pipeline. Beware though, the pipeline still seems to impersonate the user (that created the pipeline or the notebook?). I tested this by printing the access token_string that is fetched via mssparkutils in the notebook. Afterwards, I decoded the token on jwt.io and it did show that my user was the one requesting the token. It was the same behaviour whether I actively run the pipeline or the pipeline got triggered from a schedule.
end.
Obviously, it is not recommended to use secrets in clear text in your notebooks as we did with the Service Principal approach. You would actually want to store the secret and probably even the ClientID in an Azure Key Vault and then retrieve them from the notebook. Or as shown at the very end of this article, you could just get the authentication done via mssparkutils. This might be even the go to solution in Fabric Notebooks anyway. But once again, there is some impersonation going on.
appendix.
If you wanna use a Service Principal for authenticating to Fabric/Power BI, you first need to create a Service Principal in Azure. For this, go into the Azure Portal and create a new app registration. Note, it does not seem that we need to setup any scopes/API permissions.
The Tenant ID (or Directory ID) can be found in the Overview tab of your Service Principal. The Service principal ID equals the Application or Client ID:
The Service principal key is a secret you need to create under Certificates & secrets:
Also, you have to allow Service Principals to use REST API calls. You can find and enable this setting in the Admin portal in Fabric. Note, it is probably a good idea to not apply this configuration to the entire organization. It is better to use dedicated security groups for such matters.
And lastly, we need to add the Service Principal to our Fabric workspace at least as a contributor:
you saved me life mate, this is amazing
Thank you! :)