Standardization#

Skepticism#

With our central goal being to unify all ML frameworks, you would be entirely forgiven for raising an eyebrow 🤨

“You want to try and somehow unify: TensorFlow, PyTorch, JAX, NumPy, and others, all of which have strong industrial backing, huge user momentum, and significant API differences?”

Won’t adding a new “unified” framework just make the problem even worse…

https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/background/standardization/how_standards_proliferate.png?raw=true

Complimentary vs Competitive#

When considering this rather funny comic strip, it’s easy to question the feasibility of standardization in this ML space. However, there is a very important difference in the case of Ivy. Unlike A/C Chargers, Character Encodings and Instant Messaging where it’s very much one standard or the other, with Ivy this is not the case.

While Ivy does adhere to the Python Array API Standard, Ivy does not mandate adoption of the standard. Ivy is entirely complimentary to existing frameworks, each of which can and will continue to have their own syntax and call signatures. Ivy is not a replacement. Your project can have 1% Ivy code, 10% Ivy code, or 100% Ivy code. This is entirely up to you!

Do Standards Work?#

Despite making this distinction with Ivy, we do still believe that the adoption of a shared standard by each ML framework would bring huge benefits, unrelated to what we’re doing at Ivy.

Again, contrary to A/C Chargers, Character Encodings, Instant Messaging and other bumpy roads alluded to in the comic, most of the technology sector is full of successful standards. The reason we can “build” custom computers is thanks to many essential standards for the interoperability of different computer components, such as: BIOS for hardware initialization, PCIe for interfacing components on the motherboard, RAID for storage virtualization, Bluetooth for wireless data exchange, BTX for motherboard form factors and SATA for connecting host bus adapters to storage devices.

For software, HTML enables anyone to design and host a website, TCP/IP enables different nodes to communicate on a network, SMTP makes it possible to send from Gmail to Outlook, POP enables us to open this email and IEEE 754 allows us to do calculations. These are all essential standards which our modern lives depend on. Most of these standards did not arise until there was substantial innovation, growth, and usage in the relevant area, making standardization a necessity so that all parties could easily engage.

With regards to array libraries in Python, NumPy was effectively the standard until ~2015. Since then, array libraries have seen an explosion alongside innovations in Machine Learning. Given this recent time-frame, we are in a much less mature state than all of the preceding standards mentioned, most of which arose in the 70s, 80s, and 90s. An effort to standardize at this stage is completely natural, and like in all other cases mentioned, this will certainly bring huge benefits to users!

The Array API Standard#

The Consortium for Python Data API Standards are on a mission to create this shared standard. At Ivy, we support their efforts 100% and we are in the process of adopting their standard ourselves.

https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/background/standardization/consortium.png?raw=true

The consortium is lead by Quansight, an open-source-first company made up of leading open-source developers from the following organizations: PyTorch, NumPy, Pandas, SciPy, conda, dask, Apache, MXNet, ONNX, scikit-learn, Jupyter, AWS, CuPy, RAPIDS, .NET, SymPy, Ray, modin and Spyder. Other collaborators include members of: TensorFlow, JAX, Google, OctoML and einops. Further, the consortium is sponsored by LG Electronics, Microsoft, Quansight, D E Shaw and Co, TensorFlow, JAX and Intel.

https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/background/standardization/array_api_backers.png?raw=true

Together, all major ML frameworks are involved in the Array API standard in one way or another. This is a promising sign in the pursuit of unification.

Clearly, a lot of time, thought and careful attention has gone into creating the first version of the standard, such that it simplifies compatibility as much as possible for all ML frameworks.

We are very excited to be working with them on this standard, and bringing Ivy into compliance, with the hope that in due time others also follow-suit!

Round Up

Hopefully, this has given some clear motivation for why standardization in ML frameworks could be a great thing, and convinced you that we should celebrate and encourage the foundational work by the Array API Standard 🙂

Please reach out on discord if you have any questions!