Apple released two new betas for iOS 18.2 and MacOS 15.2, with some changes for Apple Intelligence features.
ChatGPT Integration
The first beta of iOS 18.2 and MacOS 15.2 presented the option to integrate Siri with ChatGPT and it was possible to sign in and use your existing ChatGPT account.
In the new beta, Settings shows your current ChatGPT subscription level once you sign in. It shows where the user is with respect to the daily ChatGPT limit based on the subscription.
It also allows the user to update to ChatGPT Plus.
Image Playground
In the third beta released on November 11th, Apple made some changes to the Image Playground application. The application takes a prompt, a picture from Photos, or a photo directly taken from the camera, to generate images in Animation and Illustration styles. Although not as capable as some of the models in this area (DALL-E from OpenAI, Stable Diffusion from Stability AI, Midjourney and so on), it is quite useful as it is completely on-device and is open to developers with a simple API allowing the use of all functionality.
In this latest release, Apple allows the user to use Appearance, instead of photos of people.
Appearance allows the user to basically create an avatar, later supported by additional text prompts, providing a richer user experience. Apart from text prompts, Apple also allows the user to use predefined input, grouped based on Themes, Costumes, Accessories and Places.
The same functionality is available through the API, so your Image Playground-based applications will also have access to Appearances.
Image Wand
Apple’s Image Wand is a feature built-in to different products and the first instance is within Notes. On the iPhone, when in Notes, you can start the Markup operation in a note with some text in it. You can then select the new editing tool that looks like a magician’s wand.
Image Wand processes the text and generates an image based on the text. In addition to the two styles we’ve already seen in Image Playground (Animation and Illustration), Image Wand brings in the Sketch style.
You should note that it does not always the right phrases from the text. It is possible to amend the image by using the text prompt on the screen asking for a description of the image. Once the user finalises the generation by pressing Done, Image Wand pastes the picture into the note.
It is possible to use Image Wand on the iPad, which also allows the user to use the Apple Pencil. The user can draw a sketch with the pencil and the generated image will then be based not only on the text, but also on the rough sketch. The same will work on notes with drawings made with a finger.
It seems like the model Image Wand uses is the same as the one Image Playground uses, except for the Sketch style that Image Wand supports.
It looks like Image Wand can not produce images of people.
Developer Libraries
Siri and App Intents
Apple’s Xcode SDKs include the App Intent, a custom action that the application exposes to the system so that it can be used by the system and OS services such as Siri. Apple has now used the same concept to enhance the types of actions that can be exposed, including those that are meaningful for Apple Intelligence. Apple has now introduced APIs for these extended App Intents which can be used along with Apple Intelligence and Siri.
Using these, the developer can enable Siri to look at content that the application is displaying on board and do something meaningful with it. Actions supported (grouped into App Intent domains) include actions to do with Books (such as asking Siri to search for something on the Internet based on a book entity that is visible on screen), Browser actions, Camera actions, Document actions, File Management actions, Journaling actions, Email actions, Photo and Video actions, Presentation actions, Spreadsheet actions, Whiteboard actions, Word Processor and Text Editing actions.
I’ve gone through the documentation and I already have some exciting ideas to include in my applications. The good thing is that these intents can be extended when Apple provides more Apple Intelligence capabilities in the future, thus easily extending the functionality your application can provide through Siri (and consequently third party AI tools such as ChatGPT and whatever else Apple might collaborate with in the future).
Image Playground
Apple has released an API for Image Playground which basically lets a developer use the full functionality within their own application.
Check my post for last week to see how the new API can be used in your own applications.