Supporting Work

Unit 2

Stable Diffusion video clips and AI generation frames:

My process in creating these animations is listed beneath these videos and screenshots.

This first video was created using the prompts: ‘Air spirit/s’ and was generated from edited green screen footage of myself reflected in a blue hue.

Thoughts on process concerning AI generation of footage of myself with the ‘Sea Nymph’ Prompt:

This second video was created using the prompts: ‘Sea nymph’ and ‘clothed sea nymph’ since the Stable Diffusion Generation images had started to generate images of myself with nipples. Some of these frames can still be seen within this video. The original footage of myself is fully clothed, even if the clothes are skintight. This is an interesting comment on the patriarchal eye that has established this dataset. As an artist, I have had to release control over aspects of the generated images that I have included in my work, and accept Stable Diffusion Deforum AI as a subject and a collaborator. The original footage of myself was edited from green screen footage and includes an image of myself mirrored and layered, with the incorporation of different pieces of aquarium footage to indicate sea life.

This third Stable Diffusion Generation Animation was generated from edited green screen footage of myself cast in a red hue with a Timelapse effect. The prompt for this animation was ‘A harpy’ and ‘A harpy kneeling’.

In the video below, it is especially interesting to have noted how AI image generators interpret text. The Stable Diffusion Deforum Generator seemed to recognise and reproduce large capital letters, but appeared not to be able to reproduce smaller, lower case letters. I really loved the new non-existent language of letters (some recognisable, others not) that came with the new footage, so I decided to include this alongside my captions in the final work; therefore embracing the generated voice and words of an AI generator.

---

Stable Diffusion Deform AI Generation Process on Google Colaboratory:

I used the video below alongside help and support from UAL technician James Stringer to use Stable Diffusion frame generation on a MacBook Air. My own personal steps and journey are discussed in more detail below the video tutorial.

How to use Deforum Stable Diffusion on a MacBook written by an anxious computational arts student.

Step one:

Follow the link in the video above to reach this google colaboratory page. Listed here:

https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqazFMaGlZSHo4UzdmZzlkMGgyRmJ4VEdxVmpGUXxBQ3Jtc0tteGg0d0lDSEVXV0huV3RPTVBKSUlZZlVZaXNSNWRoSUg4alJyb1BpUFRURzdDNnM1Rm05bk1NWEM1WU5SVW1WeDgwSURJaGNWc2hkczVjSWZFd21ERUJBUHBpY00tNVFnb1pqdTJPd0VxUy1yb3hyYw&q=https%3A%2F%2Fcolab.research.google.com%2Fgithub%2Fdeforum%2Fstable-diffusion%2Fblob%2Fmain%2FDeforum_Stable_Diffusion.ipynb&v=rvHgcOa9gDk

To reach the following page:

Step two:

Run the setup by clicking the play triangle button.

The notebook then asks permissions to access data from you Google account storage. Click run anyway.

Step three:

I have come to understand that the GPU helps access vast amounts of cloud information on datasets for deep learning. I believe it can also speed up the process for a Mac computer. I still can’t wholly wrap my mind around it.

Run it anyway.

I also ran model and output paths by clicking that play button again. A green tick should appear on each step when it’s finished loading.

At this stage, I believe I was asked to log into my Google account to allow the permissions of this notebook.

I did. Grant permissions.

Step four:

Encounter some problems.

Begin to panic. Bother the poor technician.

There was no green tick after I try to run Models and Output paths. Instead, there is an error.

I received the following error beneath Models and Output Paths:

This is the error:

37 make_grid = False #@param {type:”boolean”} 38 grid_rows = 2 #@param —> 39 outdir = get_output_folder(output_path, batch_name) 40 41 #@markdown *Init Settings* NameError: name ‘get_output_folder’ is not defined

I searched the comments of the video for answers:

I found Olivio’s next video in which he confesses to missing a step:

At the beginning of the video, Olivio recommends downloading the following sd-v1-4.ckpt Download:

https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbmpQaHJjTS1OdFhGQ2xuNjBkRS1tN0JSVkYtd3xBQ3Jtc0ttc25qSGlyMEQ1Q2I3c1VZZ0MxdjFFb09hNC0zdWVMdVJSdDFzTjBmS0RyM3NHRF9aNU1LdmhiUFdTNXhHWkRnVFVoa1I4YThfODA2YnVWeXR5VFdTZkpkZFJuaHdEQ0VkRWQzYXM0ejVxMzY5YTlJaw&q=https%3A%2F%2Fhuggingface.co%2FCompVis%2Fstable-diffusion-v-1-4-original&v=pvciyHF_w1Q

Step five:

Download the link. I asked the technician what it is. He explained it’s a set of models that can help an AI understand how to draw hands and other objects.

I uploaded the link/zip folder to my google drive.

For a while, it still did not run.

When I left University and returned to my student halls…

Somehow it worked. The green tick appeared. The models I downloaded appear in the following image, too under ‘model_checkpoint’:

Step six:

Change input on Animation Settings to ‘Video Input’ so that the AI generator can recognise the compositions, layout and hue of your original footage.

I like to set the max frames to 1000, so I can get the most out of running a session.

Then run animation by clicking the triangle play button again and waiting for the green tick of confirmation.

Step seven:

Run these existing settings.

I felt scared to change too much, in case I would ruin the effect that’s suggested by the video tutorial. Perhaps in the future, I’ll seek to experiment with these settings further and try to understand what each corresponds to more explicitly.

Step eight:

Upload the original footage/your video by dragging it to the space on the left, where the folder symbols are. Make sure the video name doesn’t have spaces as the machine can’t always read it with spaces instead of underscores.

Step nine:

Copy the link with a right click from the uploaded video and paste the link into ‘video_init_path:’ under Video Input. Now the machine can read the original footage input.

Change ‘extract_nth_frame:’ to ‘2’. I believe this means the video will generate every second frame. It helps the process move quicker but still gives you fluidity.

Run Video Input.

Step ten:

You can change the image settings from ‘512’ on W and H to ‘700’, though the AI tends not to enjoy any figure much over ‘700’.

These figures determine the size of the generation. 512 was low for my image quality, so I tried to change it in all cases to ‘712’.

Run Load Settings.

Step eleven:

Change the ‘batch_name:’ each time you do this process. It helps differentiate the generated images in their named folder.

Change the ‘seed_behaviour:’ to the ‘fixed’ option. I believe this allows each generated image to follow the composition of the one before it, to allow for frames that flow in movement when placed together.

Run ‘Save & Display Settings.’

Step twelve:

Change ‘strength:’ to the desired figure. The strength determines how much the generated video frames differs from your original footage.

The closer to ‘1.0.’ the strength is, the closer to the original footage it will be. For my own work, I discovered that ‘0.5’ was far too removed from my footage and that 0.8 was too close to my footage. My final figures that I have used within my work have varied between 0.65 and 0.7, 0.65 being more significantly deviant from my footage and 0.7 generating frames that match the footage far more closely.

Then I ran the rest of these settings. I’m not entirely sure why the ‘init_image’ and ‘mask_file:’ are included, but I imagine it helps train the AI into a certain visual style.

Step thirteen:

Make sure the ‘image_path:’ and the ‘mp4_path’ will save the frames where you want them to be in your google drive. I keep the ‘fps’ low, but it still seemed to work.

Run these settings.

Step fourteen:

In the red of the prompts section is where you can write your prompts to filter the AI’s appropriate data.

In my Stable Diffusion experiments above these instructions, I have included the individual prompts I used for each generated video.

The red text in the below image is a list of all the prompts I used to generate the footage that I have.

Run the Prompts.

Step fifteen:

Run the final step!

Then panic.

Then realise that you must wait for literal hours for every frame of each clip to be generated. I would spend entire mornings waiting.

The frames appear as they are generated in the space below the ‘Disconnect when finished’ step.

Finally, you will find yourself staring in wonder and bewilderment as each frame is generated before your eyes, every image different, and yet corresponding to the last.

Step sixteen:

Find the generated frames in your Google Drive folder. Download them all in a folder onto your computer. Open Adobe Premiere Pro. Select File, Import and then select the first generated image. It should present you with an ‘Image Sequence‘ button. Click this button, and the generated frames will finally appear as a film. Stare in wonder, amazement and horror and feel somewhat satisfied with yourself and your machine.

To see my performance and costume design process and the scripting process for my final work, including my experimentations with Chat GPT, please proceed to the next page! (page 6 below)