Tài liệu Towards adversarial attack against embedded face recognition systems

VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY FACULTY OF COMPUTER SCIENCE AND ENGINEERING BACHELOR THESIS Towards Adversarial Attack against Embedded Face Recognition Systems Major: Computer Engineering Committee: Computer Engineering Supervisors: Dr. Le Trong Nhan Assoc. Prof. Quan Thanh Tho Reviewer: Assoc. Prof. Tran Ngoc Thinh —o0o— Authors: Nguyen Minh Dang - 1752170 Nguyen Tien Anh - 1752076 Tran Minh Hieu - 1752199 Ho Chi Minh City, July 2021 ĐẠI HỌC QUỐC GIA TP.HCM ---------TRƯỜNG ĐẠI HỌC BÁCH KHOA KHOA:KH & KT Máy tính_____ BỘ MÔN: KHMT____________ CỘNG HÒA Xà HỘI CHỦ NGHĨA VIỆT NAM Độc lập - Tự do - Hạnh phúc NHIỆM VỤ LUẬN ÁN TỐT NGHIỆP Chú ý: Sinh viên phải dán tờ này vào trang nhất của bản thuyết trình HỌ VÀ TÊN: Nguyễn Minh Đăng______________________MSSV: 1752710________ NGÀNH: KTMT______________________________LỚP:_______________________ HỌ VÀ TÊN: Trần Minh Hiếu_________________________MSSV: 1752199________ NGÀNH: KTMT______________________________LỚP:_______________________ HỌ VÀ TÊN: Nguyễn Tiến Anh________________________MSSV: 1752076________ NGÀNH: KTMT______________________________LỚP:_______________________ 1. Đầu đề luận án: Towards Adversarial Attack against Embedded Face Recognition Systems _____________________________________________________________________________ _____________________________________________________________________________ 2. Nhiệm vụ (yêu cầu về nội dung và số liệu ban đầu): ✔ Investigate face authentication techniques ✔ Research and design the desired system based on NVIDIA Jetson Nano Developer Kit ✔ Research and propose an approach to apply adversarial attack technique to prevent attacker to fool the system ✔ Implement a prototype and evaluate the performance 3. Ngày giao nhiệm vụ luận án: 4. Ngày hoàn thành nhiệm vụ: 5. Họ tên giảng viên hướng dẫn: Phần hướng dẫn: 1) Lê Trọng Nhân 2) Quản Thành Thơ 3)____________________________________________________________________________ Nội dung và yêu cầu LVTN đã được thông qua Bộ môn. Ngày ........ tháng ......... năm .......... CHỦ NHIỆM BỘ MÔN GIẢNG VIÊN HƯỚNG DẪN CHÍNH (Ký và ghi rõ họ tên) (Ký và ghi rõ họ tên) PGS.TS. Quản Thành Thơ PHẦN DÀNH CHO KHOA, BỘ MÔN: Người duyệt (chấm sơ bộ):_________________________ Đơn vị:_________________________________________ Ngày bảo vệ:____________________________________ Điểm tổng kết:___________________________________ Nơi lưu trữ luận án:_______________________________ TRƯỜNG ĐẠI HỌC BÁCH KHOA KHOA KH & KT MÁY TÍNH CỘNG HÒA Xà HỘI CHỦ NGHĨA VIỆT NAM Độc lập - Tự do - Hạnh phúc ---------------------------Ngày tháng năm PHIẾU CHẤM BẢO VỆ LVTN (Dành cho người hướng dẫn/phản biện) 1. Họ và tên SV: Nguyễn Minh Đăng MSSV: 1752710 Họ và tên SV: Trần Minh Hiếu MSSV: 1752199 Họ và tên SV: Nguyễn Tiến Anh MSSV: 1752076 Ngành (chuyên ngành): KTMT Ngành (chuyên ngành): KTMT Ngành (chuyên ngành): KTMT 2. Đề tài: Towards Adversarial Attack against Embedded Face Recognition Systems 3. Họ tên người hướng dẫn/phản biện: PGS.TS. Quản Thành Thơ 4. Tổng quát về bản thuyết minh: Số trang: Số chương: Số bảng số liệu Số hình vẽ: Số tài liệu tham khảo: Phần mềm tính toán: Hiện vật (sản phẩm) 5. Tổng quát về các bản vẽ: - Số bản vẽ: Bản A1: Bản A2: Khổ khác: - Số bản vẽ vẽ tay Số bản vẽ trên máy tính: 6. Những ưu điểm chính của LVTN: - - The students addressed an emerging security problem in the area of face recognition. The solution proposed by students include a selection of suitable hardware device and especially an AI approach for black-box adversarial attack, whose performance overcomes the current state-of-the-art results. To achieve this, the students has conducted a very insightful literature review, gradually elaborated their suggested architecture and successfully implemented their models with impressive performance The work in this thesis has been publish in two papers, one in a student scientific conference and especially in prestigious international conference, whose proceedings are published by Springer. This should illustrate excellent result of the students’ work. 7. Những thiếu sót chính của LVTN: 8. Đề nghị: Được bảo vệ  Bổ sung thêm để bảo vệ  9. 3 câu hỏi SV phải trả lời trước Hội đồng: a. 10. Đánh giá chung (bằng chữ: giỏi, khá, TB): Không được bảo vệ  Điểm : 10 /10 Ký tên (ghi rõ họ tên) PGS.TS. Quản Thành Thơ 75ѬӠ1*ĈҤ,+Ӑ&%È&+.+2$ KHOA KH & KT MÁY TÍNH &Ӝ1*+Ñ$;­+Ӝ,&+Ӫ1*+Ƭ$9,ӊ71$0 ĈӝFOұS- 7ӵGR- +ҥQKSK~F ---------------------------Ngày 08 tháng 08 QăP21 3+,ӂ8&+Ҩ0%Ҧ29ӊ/971 'jQKFKRQJ˱ͥLSK̫QEL͏Q) +ӑYjWrQ69 Nguyen Minh Dang +ӑYjWrQ69Nguyen Tien Anh +ӑYjWrQ69Tran Minh Hieu MSSV: 1752170 MSSV: 1752076 MSSV: 1752199 Ngành (chuyên ngành): .ӻWKXұW Máy Tính ĈӅWjL Towards Adversarial Attack against Embedded Face Recognition Systems +ӑWrQQJѭӡLSKҧQELӋQ: Assoc. Prof. Dr. 7UҫQ1JӑF7KӏQK 7әQJTXiWYӅEҧQWKX\ӃWPLQK 6ӕWUDQJ 83 6ӕFKѭѫQJ 6 6ӕEҧQJVӕOLӋX:10 6ӕKuQKYӁ 35 6ӕWjLOLӋXWKDPNKҧR 104 3KҫQPӅPWtQKWRiQ +LӋQYұWVҧQSKҭP: 01 Adversarial attack system on Jetson Nano 7әQJTXiWYӅFiFEҧQYӁ - 6ӕEҧQYӁ %ҧQ$ %ҧQ$ .KәNKiF - 6ӕEҧQYӁYӁWD\ 6ӕEҧQYӁWUrQPi\WtQK 1KӳQJѭXÿLӇPFKtQKFӫD/971 a. The students successfully proposed a new attack algorithm on face recognition systems that works reliably in the physical world without requiring any knowledge about the victim model. b. Their methodology had been evaluated on various model architectures and training losses. As compared the baseline, the attack success rates of their system are far better. c. They deployed a face recognition system on a Jetson Nano and proved it to work well. d. 01 paper has been accepted by The 4th International Conference on Multimedia Analysis and Pattern Recognition (MAPR 2021). 1KӳQJWKLӃXVyWFKtQKFӫD/971 The proposed methodology is only applied for global physical attacks. The students should extend it for both global & local physical attacks. ĈӅQJKӏĈѭӧFEҧRYӋ; %әVXQJWKrPÿӇEҧRYӋ† 9. 2 FkXKӓL69SKҧLWUҧOӡLWUѭӟF+ӝLÿӗQJ .K{QJÿѭӧFEҧRYӋ† a. Based on your proposed methodology, how can you help mitigate or avoid adversarial attacks against Face Recognition Systems? b. Most related works use PCs to deploy their systems, why your system is deployed on a Jetson Nano with low performance. ĈiQKJLiFKXQJEҵQJFKӳJLӓLNKi7% Very Good ĈLӇP9.5 /10 Ký tên (ghi rõ KӑWrQ 7UҫQ1JӑF7KӏQK Declaration of Authenticity We hereby declare that this thesis titled "Towards Adversarial Attack against Embedded Face Recognition Systems" and the work presented in it are our own. We confirm that: • This work was done wholly or mainly while in candidature for a degree at this University. • Where any part of this thesis has previously been submitted for a degree or any other qualification at this University or any other institution, this has been clearly stated. • Where we have consulted the published work of others, this is always clearly attributed. • Where we have quoted from the work of others, the source is always given. With the exception of such quotations, this thesis is entirely our own work. • We have acknowledged all main sources of help. • Where the thesis is based on work done by ourselves jointly with others, we have made clear exactly what was done by others and what we have contributed ourselves. Ho Chi Minh City, July 2021 i Acknowledgement Firstly, we would like to show our deepest gratitude to our supervisors, Professor Quan Thanh Tho and Dr. Le Trong Nhan, for their invaluable time, patience, and warm support. They have spent so much effort guiding us, and their insightful feedback has helped us realize the weaknesses in our work. Furthermore, their enthusiasm has been an encouragement to help us move forward during the difficult stage of our research. Without the help from them, this thesis could not have come to reality. Secondly, we want to thank all the lecturers for all the knowledge and skills they provided us in the past four years. Thank HCMC University of Technology and the Faculty of Computer Science and Engineering for creating such a wonderful incubating environment that has helped us grow as students as well as individuals. Finally yet importantly, we thank our beloved friends and family for their immense amount of love, support, and encouragement throughout the years. It has been an incredible journey, we wish you all good health and happiness in life. Nguyen Minh Dang, Nguyen Tien Anh, Tran Minh Hieu ii Abstract Numerous studies have shown that deep neural networks (DNNs) are vulnerable to adversarial examples - malicious inputs that are carefully crafted to cause a model to misclassify. This phenomenon raises a serious concern, especially for Deep learning-based security-critical systems such as face recognition. However, most of the studies on the adversarial vulnerability of DNNs have only considered the ideal scenarios (e.g., they assume the attackers have perfect information about the victim model or the attack is performed in the digital domain). As a result, these methods often poorly (or even impossible to) transfer to the real world and hamper future studies on defense mechanisms against realworld attacks. To address this issue, we propose a novel physically transferable attack on deep face recognition systems. Our method can work in the physical world settings without requiring any knowledge about the victim model. Our extensive experiments on various model architectures and training losses show non-trivial results and give rise to some interesting observations that can be a potential research direction in the future to improve the robustness of models against adversarial attacks. iii Contents List of Figures vii List of Tables ix List of Notations x 1 Introduction 1 1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Thesis Scopes and Objectives . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Our contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 Background Knowledge 2.1 7 Deep Learning and Neural Networks . . . . . . . . . . . . . . . . . . . . . 8 2.1.1 Artificial Neural Networks . . . . . . . . . . . . . . . . . . . . . . . 8 2.1.2 Convolutional Neural Networks . . . . . . . . . . . . . . . . . . . . 10 2.2 Optimization Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.3 Face Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.4 Adversarial Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.5 2.4.1 Adversarial Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.4.2 Properties of Adversarial Examples . . . . . . . . . . . . . . . . . . 16 2.4.3 A Taxonomy of Adversarial Attacks . . . . . . . . . . . . . . . . . . 17 2.4.4 Generating Adversarial Examples . . . . . . . . . . . . . . . . . . . 21 Jetson Nano . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.5.1 Developer kit and Hardware . . . . . . . . . . . . . . . . . . . . . . 22 2.5.2 JetPack and libraries . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3 Literature Review 3.1 28 Black-box adversarial attacks . . . . . . . . . . . . . . . . . . . . . . . . . 29 iv 3.1.1 Decision-based adversarial attacks: Reliable attacks against blackbox machine learning models. . . . . . . . . . . . . . . . . . . . . . 29 3.1.2 Efficient Decision-based Black-box Adversarial Attacks on Face Recognition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.2 Adversarial attacks in the physical world . . . . . . . . . . . . . . . . . . . 32 3.2.1 Accessorize to a Crime: Real and Stealthy Attacks on State-of-theArt Face Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.2.2 AdvHat: Real-world adversarial attack on ArcFace Face ID system 4 Methodology 34 35 4.1 Threat Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.2 Baseline Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.3 From Digital to Physical World Attack . . . . . . . . . . . . . . . . . . . . 38 4.4 Enhancing the Transferability of Transfer-based Attacks . . . . . . . . . . 39 5 Experiments 5.1 5.2 41 Experimental Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.1.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.1.2 Pre-trained Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 5.1.3 Evaluation Metric . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 5.1.4 Physical Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 5.2.1 Attack success rates in the physical world . . . . . . . . . . . . . . 46 5.2.2 Performance comparisons between digital and physical world . . . . 49 5.2.3 Sensitivity to epsilon and the number of ensemble models . . . . . . 50 5.2.4 Extended experiments on local adversarial attacks . . . . . . . . . . 51 5.2.5 Evaluation on NVIDIA Jetson Nano Embedded System . . . . . . . 55 6 Conclusion and Future Works 57 Bibliography 67 Appendices 68 A FaceX-Zoo and LFW Dataset 69 A.1 Preparation and dependencies . . . . . . . . . . . . . . . . . . . . . . . . . 69 v A.2 Face cropping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 A.3 Pre-trained models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 A.4 LFW Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 B Deploying Face Recognition on NVIDIA Jetson Nano 72 B.1 Prerequisite and installation guide . . . . . . . . . . . . . . . . . . . . . . . 72 B.1.1 Hardware requirements . . . . . . . . . . . . . . . . . . . . . . . . . 72 B.1.2 Software dependencies . . . . . . . . . . . . . . . . . . . . . . . . . 73 B.2 System descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 B.2.1 Face Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 B.2.2 Face Representation . . . . . . . . . . . . . . . . . . . . . . . . . . 75 B.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 C Published Paper 78 vi List of Figures 1.1 An overview of our work. Given a black-box face recognition model, a pair of source and target images, we aim to generate an adversarial noise that, when added to the source image, causes the model to misclassify them as belonging to the same identity. . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1 A simple deep neural network with 2 hidden layers. . . . . . . . . . . . . . 8 2.2 Popular activation functions . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3 Architecture of the LeNet-5 network 2.4 Gradient Descent in the physical view . . . . . . . . . . . . . . . . . . . . . 12 2.5 A standard pipeline of an end-to-end face recognition system (Du et al., . . . . . . . . . . . . . . . . . . . . . 10 2021). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.6 Adversarial examples (Goodfellow et al., 2015) . . . . . . . . . . . . . . . . 15 2.7 An illustration of accessible components of the target model for each of the three threat models. A white-box threat model assumes access to the whole model; a score-based threat model assumes access to the output layer; a decision-based threat model assumes access to the predicted label alone (J. Chen et al., 2020). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.8 Adversarial examples in the physical world . . . . . . . . . . . . . . . . . . 19 2.9 An example of local adversarial attack (Brown et al., 2018) . . . . . . . . . 20 2.10 Jetson Nano Developer Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.11 Developer kit module and carrier board (Nvidia, 2020) . . . . . . . . . . . 23 2.12 TensorRT workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.1 Basic intuition of the Boundary attack . . . . . . . . . . . . . . . . . . . . 30 3.2 An example of the targeted attack, along with the number of model calls. . 30 3.3 An example of the dodging(untargeted) attack and impersonate(targeted) attack. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.4 Attackers with the adversarial eye-frames. . . . . . . . . . . . . . . . . . . 33 vii 3.5 Adversarial stickers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 5.1 Image preprocessing for the LFW dataset. The first row shows the original images, and the second row shows the images after preprocessing. . . . . . 42 5.2 An example of our rebroadcast process. We display the generated adversarial examples onto a monitor, then capture them using another device. . 45 5.3 Digital and Physical ASR backbone-wise in global setting. . . . . . . . . . 49 5.4 Digital and Physical ASR head-wise in global setting. . . . . . . . . . . . . 49 5.5 ASR with respect to ε values in global setting. . . . . . . . . . . . . . . . . 50 5.6 ASR with respect to the number ensemble models in global setting. . . . . 50 5.7 Global and local perturbations of an image. . . . . . . . . . . . . . . . . . 51 5.8 Eye-glasses shape. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 5.9 Global and Local ASR backbone-wise in digital setting. . . . . . . . . . . . 52 5.10 Global and Local ASR backbone-wise in physical setting. . . . . . . . . . . 53 5.11 Global and Local ASR head-wise in digital setting. . . . . . . . . . . . . . 53 5.12 Global and Local ASR head-wise in physical setting. . . . . . . . . . . . . 54 5.13 ASR with respect to ε. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 5.14 ASR with respect to the number of ensemble models. . . . . . . . . . . . . 54 A.1 FaceX-Zoo on Github. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 A.2 Backbone-wise models in FaceX-Zoo. . . . . . . . . . . . . . . . . . . . . . 70 A.3 Head-wise models in FaceX-Zoo. . . . . . . . . . . . . . . . . . . . . . . . . 71 B.1 Pipeline of the cascaded framework that includes three-stage multi-task deep convolutional networks. . . . . . . . . . . . . . . . . . . . . . . . . . . 74 B.2 FaceNet high-level model structure. . . . . . . . . . . . . . . . . . . . . . . 75 B.3 Triplet loss intuition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 viii List of Tables 2.1 Notable loss functions for Face Representation. . . . . . . . . . . . . . . . . 15 2.2 Nvidia Jetson Nano module technical specification . . . . . . . . . . . . . . 23 2.3 Nvidia Jetson Nano carrier board components . . . . . . . . . . . . . . . . 24 5.1 Model accuracy evaluates on the LFW test set and their corresponding best cosine similarity threshold. . . . . . . . . . . . . . . . . . . . . . . . . 44 5.2 Backbone-wise ASR results. ASR Baseline is the mean ASR when attack using the baseline method. ASR with M-E is the mean ASR when attack using our method without including Diverse Input method. ASR with M-E-DI is the mean ASR when attack using our method. . . . . . . . . . 47 5.3 Head-wise ASR results. ASR Baseline is the mean ASR when attack using the baseline method. ASR with M-E is the mean ASR when attack using our method without including Diverse Input method. ASR with M-E-DI is the mean ASR when attack using our method. . . . . . . . . . 47 5.4 Head-wise ASR results. ASR Baseline is the mean ASR when attack using the baseline method. ASR with M-E is the mean ASR when attack using our method without including Diverse Input method. ASR with M-E-DI is the mean ASR when attack using our method. . . . . . . . . . 48 5.5 Source - Target and Adversarial - Target l2 distance in global adversarial attack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.6 Source - Target and Adversarial - Target l2 distance in local adversarial attack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 B.1 Libraries/packages specifications. . . . . . . . . . . . . . . . . . . . . . . . 73 B.2 The deployed system evaluation. . . . . . . . . . . . . . . . . . . . . . . . . 77 ix List of Notations A Matrix Aij Matrix indexed for some purpose Ai Matrix indexed for some purpose Aij Matrix indexed for some purpose An Matrix indexed for some purpose or the n-th power of a square matrix A−1 The inverse matrix of the matrix A A+ The pseudo inverse matrix of the matrix A A1/2 The square root of a matrix (if unique), not elementwise (A)ij The (i, j)-th entry of the matrix A Aij The (i, j)-th entry of the matrix A [A]ij The ij -submatrix, i.e A with the i-th row and j-th column deleted a Vector (column-vector) ai Vector indexed for some purpose ai The i-th element of the vector a a Scalar det(A) Determinant of A Tr(A) Trace of the matrix A diag(A) Diagonal matrix of the matrix A, i.e. (diag(A))ij = δij Aij eig(A) Eigenvalues of the matrix A vec(A) The vector-version of the matrix A kAk Matrix norm (subscript if any denotes what norm) x Chapter 1 Introduction In this chapter, we give an overview of our thesis, define the thesis scopes, objectives, and summarize our contributions. 1 1.1 Overview Deep learning is a branch of machine learning in which learning models are made up of multiple layers. The advent of deep learning has created numerous breakthroughs in handling problems where traditional machine learning techniques perform poorly. Computer vision tasks such as image classification (Dosovitskiy et al., 2020; Foret et al., 2020), object detection (Ghiasi et al., 2020; C.-Y. Wang et al., 2020), face recognition (Deng, Guo, Zhou, et al., 2019; Schroff et al., 2015), semantic segmentation (Mohan et al., 2020; Yuan et al., 2020), and natural language processing tasks such as semantic analysis (Lan et al., 2020; Raffel et al., 2020), question answering (Joshi et al., 2020; Yang et al., 2020), machine translation (Edunov et al., 2018; Zhu et al., 2020). Specifically, deep learning is the dominant approach in many real-life applications such as virtual assistants (Google Assistant, Alexa, Siri), machine translation tools (Google Translate and IBM Watson Language Translator), autonomous vehicles (Tesla, Audi, and BMW), or corporate facial recognition systems to identify employees. In recent years, deep learning has also been applied in highly complicated tasks such as analyzing the potential of drug molecules (Ma et al., 2015), reconstruction of brain circuits (Helmstaedter et al., 2013), analyzing particle accelerator data (de Seixas T. Ciodaro et al., 2012), effects of mutations in DNA (Lee et al., 2015). In addition, with the improvement in the computation power of hardware such as GPU and TPU, the process of training and inference has become significantly simpler and faster. The introduction of the Convolutional Neural Network (CNN) revolutionized deep learning, especially in computer vision applications. Specifically, for applications such as object detection, image classification, face recognition, semantic segmentation. the CNN network has increased performance dramatically and has become the dominant approach. The CNN architecture is similar to the neurons’ communication pattern in the human brain and was influenced by the Visual Cortex organization. Individual neurons respond only in a small area of the visual field known as the receptive field to stimuli. To cover the entire visual region, a range of such fields overlap. By applying appropriate filters, a CNN can successfully capture the spatial dependencies in an image. The structure of a CNN network can model the image data set well thanks to the reduction in the number of parameters involved and reusability of weights (Khan et al., 2020). In other words, the network can be trained to understand the sophistication of the image better. 2 In the aforementioned computer vision applications, face recognition, which is the prominent biometric technique for identity authentication, has been widely used in today’s fields such as military, finance, and information security. Face recognition has long been a research topic in the machine learning community around the world since the 1990s. At that time, with traditional approaches such as holistic learning (Belhumeur et al., 1997; Moghaddam et al., 1998), local handcraft (Chengjun Liu et al., 2002; Wenchao Zhang et al., 2005), shallow learning (Cao et al., 2010; Lei et al., 2014), the achieved accuracy is not high due to many reasons such as a lack of distinctiveness and compactness, limitation on robustness against the complex nonlinear facial appearance variations. However, thanks to deep learning and, in particular, CNN, the accuracy has improved remarkably, and is comparable with human performance (Parkhi et al., 2015; Taigman et al., 2014). In recent years, several studies have shown that deep neural networks are vulnerable to adversarial examples - malicious inputs that are carefully crafted to force the models to make erroneous predictions (Goodfellow et al., 2015; Szegedy et al., 2014). Moreover, some adversarial examples are almost identical to the original images, making it difficult to discern visually. This raises a serious security concern especially when deep learning have been widespread in everyday life applications. There have been several works study the adversarial vulnerability of deep face recognition system (Erdogmus et al., 2013; Komkov et al., 2019). However, most of them have only considered the ideal scenarios. For example, (Sharif et al., 2016a) assumes that the attackers have perfect knowledge about the victim model, including its parameters, architecture, and gradients. This type of attack is often classified as white-box attacks. On the other hand, (Dong et al., 2019) proposed black-box attacks that do not require any prior knowledge about the victim model, but they assume that the attack takes place in the digital domain - where inputs are fed directly into the model (e.g., via an API). This type of attack is also known as digital attacks. Adversarial attacks in the white-box or digital settings are relatively simple to achieve, but they are often ineffective or even impossible to be applied in real-world settings. Firstly, attackers do not often have permission to obtain the model’s internal configurations. Most of the time, only the label predicted by the model is accessible to the attacker. Thus, attack in such limited settings 3 Figure 1.1: An overview of our work. Given a black-box face recognition model, a pair of source and target images, we aim to generate an adversarial noise that, when added to the source image, causes the model to misclassify them as belonging to the same identity. is more challenging to achieve. Secondly, real-world systems do not always provide an open-access API, and the only way to attack a model is likely via a sensory input device (e.g., a camera). In the second case, the malicious input has to undergo two processes: (1) digital-to-analog: attackers convert the generated adversarial example to the physical world; then (2) analog-to-digital : the model’s sensory input device reconstructs the physical adversarial example to the digital domain. The above 2-step process is often referred to as image rebroadcasting (Agarwal et al., 2018), and it has been shown to diminish the effectiveness of adversarial examples due to environmental factors such as a change in lighting, contrast, and distance to the camera (Athalye et al., 2018). In this work, we aim to investigate the vulnerability of deep face recognition systems in a more realistic scenario. That is, we assume that (1) the attackers only have access to the model’s hard-labels outputs without any knowledge of its internal configurations; (2) the attack takes place in the physical domain. Figure 1.1 illustrates what we aim to achieve. Given a black-box face recognition model, a pair of images from different people, we aim to generate an adversarial noise that causes the model to misclassify them as belonging to the same identity. To tackle this problem, we propose a novel physical transferable attack method that can work without prior knowledge about the victim model, and subsequently, the produced adversarial examples remain effective in the physical domain. Most importantly, our method is efficient since it does not require any query to the victim model to generate adversarial examples. We perform extensive experiments on the Labeled Face in 4 the Wild (LFW) dataset (G. B. Huang et al., 2007a), one of the most popular benchmark datasets of face recognition tasks. In addition, we evaluate our method on various pretrained state-of-the-art face recognition models with different architectures and training losses. The pre-trained models are provided from the open-source repository FaceX-Zoo (J. Wang et al., 2021). Although studying new attack methods seems dangerous and harmful, we argue that it has important scientific value. Firstly, it helps us gain valuable insight into how DNNs works (Ilyas et al., 2019; Schmidt et al., 2018). Secondly, it serves as a base for further studies on defense strategies to make deep face recognition systems more secure. Finally yet interestingly, adversarial attacks also have practical real-world applications, for example, in enhancing the security of CAPTCHAs (Shao et al., 2021) or protecting individual’s privacy (Wu et al., 2020). 1.2 Thesis Scopes and Objectives In this work, we aim to propose an adversarial attack algorithm against face recognition systems in targeted physical black-box setting. The face recognition systems we consider in our thesis are state-of-the-art Deep Learning-based models trained with standard training procedure in H. Wang et al., 2018. This work does not include face recognition systems equipped with an anti-spoofing module. We also do not aim to propose a defense mechanism against our attack since it is beyond the scope of our interest and adversarial defense is currently one of the most challenging unsolved problem (Carlini et al., 2019). For concreteness, the goals of our thesis includes: • Propose a targeted physical black-box attack algorithm on face recognition systems. • Evaluate the proposed attack on various model architectures and training losses. • Demonstrate the effectiveness of the proposed attack on an embedded face recognition system. 5