Scribd Logo
Upload

Welcome to Scribd!

  • Snapchat Object Recognition Based Photo Filters Patent Application

    Uploaded by

    Lara
    47K views29 pages
    The patent application filed in January 9, 2015 and published by the US Patent and Trademark Office on July 14, 2016.

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    The patent application filed in January 9, 2015 and published by the US Patent and Trademark Office on July 14, 2016.

    Copyright:

    © All Rights Reserved
    Download as PDF or read online from Scribd
    Flag for inappropriate content
    47K views29 pages

    Snapchat Object Recognition Based Photo Filters Patent Application

    Uploaded by

    Lara
    The patent application filed in January 9, 2015 and published by the US Patent and Trademark Office on July 14, 2016.

    Copyright:

    © All Rights Reserved
    Download as PDF or read online from Scribd
    Flag for inappropriate content
    of 29
    cu») United States US 201602035864 1 cz) Patent Application Publication — co) Pub. No.: US 2016/0203586 A1 Chang et al. (64) OBJECT RECOGNITION BASED PHOTO FILTERS (71) Applicant: Snapehat, In. Veniee, CA (US) (72) Inveators: Sheldon Chang, Venice, CA (US): ‘Chamal Samaranayake, Venice. CA (US); Timothy Michael Sehn, Marina Del Rey, CA (US); Rong Yan, Marina Del Rey, CA(US) (21) Appl. Nos 141593,065 (22) Filed: Jan. 9, 2018 Publication Classification on) (2005.01) (2006.01) 06K 9148 (43) Pub. Date: Jul. 14, 2016 (52) US. CBC... GOBT $/00 (2013.01); GO6K 9/48 2013.01) on ABSTRACT ‘Systems and methods for generating and distributing photo filters are deseribed. A photo filter publication application mn the filter data, The photo filter iss fon ofthe object eriteri. A photo filter engine sthataclientdevice has taken a photograph The photo filter engine then provides the photo filter the elent ‘Sevice based on the photograph including an object that sat- istes the object criteria, The photo filter may then be dis- played as an option on a wer interface of the client device. The object exiteria may include associations between an object and a source of image dat, for example, a brand of a ‘mereiant in which caso the associated photo filer may include images associated withthe brand of the merchant. USER-BASED PHOTO FILTER PUBLICATION MODULE 314 USER-BASED FILTER DATA UPLOAD MODULE 402 USER-BASED OBJECT CRITERIA UPLOAD MODULE 404 USER-BASED VIDEO CREATION MODULE 406 USER-BASED PUBLICATION ENGINE 408 Patent Application Publication Jul. 14, 2016 Sheet 1 of 17 US 2016/0203586 A1 CLIENT DEVICE 110 CLIENT DEVICE 412 PROGRAMMATIC. PROGRAMMATIC. CLIENT 106 CLIENT 105, PHOTO APPLICATION PHOTO APPLICATION Ed ior eo NETWORK (E.G., INTERNET) 104 APISERVER (PROGRAMMATIC. INTERFACE) 114 i ‘APPLICATION SERVER 118 I 1 I ' I I I ' ' I MESSAGING PHOTO FILTER I APPLICATION ft) APPLICATION I 120 122 I I I I ' ! I I I I I i DATABASE SERVER 124 — +>} DATABASE(S) 126 Patent Application Publication Jul. 14, 2016 Sheet 2 0f 17 US 2016/0203586 A1 MESSAGING APPLICATION 120 PHOTO RECEIVER Perry MODLATE 20a ‘APPLICATION INTERFACE 204 EPHEMERAL MESSAGE MESSAGE GENERATOR STORAGE MODULE 210 MODULE 206 EPHEMERAL MESSAGE ACCESS MODULE 208 FIG. 2 Patent Application Publication Jul. 14, 2016 Sheet 3 0f 17 US 2016/0203586 A1 PHOTO FILTER APPLICATION 122 PHOTO FILTER PUBLICATION MODULE 304 USER-BASED MERCHANT-BASED PUBLICATION MODULE PUBLICATION MODULE 314 316 PHOTO FILTER ENGINE 306. PHOTO FILTER OBJECT MERCHANT-BASED PRIORITY, RECOGNITION PHOTO FILTER MODULE 318 MODULE 320 MODULE 322 COLLECTION COUNT, MODULE MODULE: 324 326 Patent Application Publication Jul. 14, 2016 Sheet 4 of 17 US 2016/0203586 A1 USER-BASED PHOTO FILTER PUBLICATION MODULE. 34 USER-BASED FILTER DATA UPLOAD MODULE 402 USER-BASED OBJECT CRITERIA UPLOAD MODULE 404 USER-BASED >| PUBLICATION ENGINE 408 USER-BASED VIDEO CREATION MODULE >| 406 FIG. 4A US 2016/0203586 AI Jul. 14,2016 Sheet 5 of 17 Patent Application Publication oer aly ony vip ap O14 ew WIV YLT VINALINO 4193°80 sais 3OvWIGvOTdN S73CGOW 1o3rgo avordn \ oly Zip Patent Application Publication Jul. 14, 2016 Sheet 6 of 17 US 2016/0203586 A1 = gS & & ° \ aise ees) = I { 1 | w — eo] [2] fe ne Wo = we] JEG] | eS E = g} |3 i Z : FIG. 4C 410 412 Patent Application Publication Jul. 14, 2016 Sheet 7 0f17 US 2016/0203586 A1 OBJECT RECOGNITION IN | PHOTOGRAPH 440 ) C- Qh rae Sey KING KONG! a PHOTO CONTENT ~—~_| + FIG. 4D Patent Application Publication Jul. 14, 2016 Sheet 8 of 17 US 2016/0203586 A1 MERCHANT-BASED PHOTO FILTER PUBLICATION MODULE 316 MERCHANT-BASED FILTER DATA UPLOAD + MODULE 502 MERCHANT-BASED OBJECT CRITERIA >| eee MERCHANT-BASED PUBLICATION MERCHANT-BASED ENGINE 510 VIDEO CREATION >| MODULE 506 MERCHANT-BASED BIDDING MODULE. + 508 COMMON OBJECT MERCHANT A PHOTO FILTER | CRITERIA 516 OBJECT CRITERIA 512 = MERCHANT 8 PHOTO FILTER OBJECT CRITERIA 514 FIG. 5B Patent Application Publication Jul. 14, 2016 Sheet 9 of 17 US 2016/0203586 A1 eo a + 550 MERCHANT A OBJECT CRITERIA i kK. ' 1 MON OBJECT! MERCHANT B! er OBJECT TYPE: COFFEE | ' 580 ! ' ' ' ' ' ' I i ' ' | Coffee Brand A ' ' ' \ ' ' ' \ PHOTO CONTENT 570 —“~__-” FIG. 5C Patent Application Publication Jul. 14, 2016 Sheet 10 of 17 US 2016/0203586 A1 1200 1210, 610 | 4 Uy 620 20% off 7 off coupon en 4.uses remaining} Coffee TAP HERE TO 640 House PRINT = “J Oy PHOTO CONTENT ele, FIG. 6A Patent Application Publication 650 1200 660 Jul. 14,2016 Sheet 11 of 17 1210, ) eee, LI fl souP Collect two more Deli a free sandwich PHOTO CONTENT FIG. 6B US 2016/0203586 AI Patent Application Publication 700 \ BEGIN Y Jul. 14,2016 Sheet 12 of 17 US 2016/0203586 AI RECEIVE FILTER DATA AND OBJECT CRITERIA FROM A. FIRST CLIENT DEVICE, Ton Y GENERATE PHOTO FILTER. ‘ASSOCIATED WITH THE UPLOADED OBJECT CRITERIA 704 i RECOGNIZE OBJECT IN PHOTOGRAPH TAKEN BY A SECOND CLIENT DEVICE 706 i OBJECT IN PHOTOGRAPH SATISIFES OBJECT CRITERIA? 108 T es PROVIDE PHOTO FILTER TO THE SECOND CLIENT DEVICE no END FIG.7 No Patent Application Publication Jul. 14,2016 Sheet 13 of 17 800 \ BEGIN ¥ RECEIVE FILTER DATA A FROM MERCHANT @ WITH OBJECT CRITERIA X AND BID AMOUNT A. 02 v RECEIVE FILTER DATA B FROM MERCHANT B WITH OBJECT CRITERIA X AND BID AMOUNT B US 2016/0203586 AI 04 ¥ ves No BID AMOUNT A> BID AMOUNT B 7 808 v ¥ ‘GENERATE PHOTO FILTER A (GENERATE PHOT FILTER B BASED ON FILTER DATA A FOR BASED ON FILTER DATA B FOR OBJECT CRITERIA X OBJECT CRITERIA X 08 aie v v ‘SUPPLY PHOTO FILTER ATO ‘SUPPLY PHOTO FILTER B TO CLIENT DEVICES BASD ON, CLIENT DEVICES BASD ON ‘SATIFACTRON OF OBJECT SATIFACTRON OF OBJECT CRITERIA X 810 CRITERIA X £10 END FIG.8 Patent Application Publication Jul. 14, 2016 Sheet 14 of 17 US 2016/0203586 A1 900 “™~ BEGIN Y NEW PHOTO FILTER RECEIVED BY A CLIENT DEVICE 902 Y No IS THE PHOTO FILTER THE REQUIRED TYPE? 904 YES ¥ ‘ADD PHOTO FILTER TO COLLECTION (OF CLIENT DEVICE 906 * NO IS THE COLLECTION THE REQUIRED SIZE? 908 YES ¥ PROVIDE NEW PREMIUM PHOTO FILTER TO CLIENT DEVICE g10 END <——__1 FIG.9 Patent Application Publication Jul. 14, 2016 Sheet 15 of 17 US 2016/0203586 A1 1000- o™ BEGIN Y NEW PHOTO TAKEN BY ‘A CLIENT DEVICE 1002 No PHOTO INCLUDES OBJECT OF REQUIRED TYPE? 1904 | yes INGREASE COUNT ASSOCIATED WITH THE OBJECT TYPE 1906 1 IS THE COUNT THE REQUIRED SIZE? 1008 | ves ‘ADJUST CONTENT OF PHOTO FILTER ‘ASSOCIATED WITH OBJECT TYPE 910 I END \}¢——_______ No FIG. 10 Patent Application Publication Jul. 14, 2016 Sheet 16 of 17 US 2016/0203586 A1 iv 100 PROCESSOR VIDEO fun aa DISPLAY a1za—fnsTRUCTIONS MAIN MEMORY ALPHANUMERIC INPUT Device | ~ +142 suza—{insTRUCTIONS| 108 _) stare ance) MEMORY UI NAVIGATION] DEVICE 14 DRIVE UNIT COMPUTER” ]}-1116 NETWORK READABLE ue] INTERFACE Meo if 122 a24 SIGNAL 1126 GENERATION F-11183 DEVICE FIG. 11 Patent Application Publication Jul, 14, 2016 Sheet 17 of 17 1200. ™~ US 2016/0203586 AI 1216 MOBILE DEVICE DISPLAY 1210 GPS RECEIVER 1218, ‘TRANSCEIVER 1214 t MEMORY >| 1204 PROCESSOR 1202 os 1206 | APPLICATIONS 1208 VO DEVICES 212 FIG. 12 US 2016/0203586 AI OBJECT RECOGNITION BASED PHOTO FILTERS TECHNICAL FIELD [0001] The subject matter disclosed herein generally relates to generating and distributing photo filters. More spe- ‘lial, the present disclosure axidesses systems and meth: ‘ods for preseting photo filerson a user interface of a mobile ‘device based on recognizing, ina photograph taken with the ‘mobile device, objects that satisly specified object eter BACKGROUND 10002] The number of digital photographs taken with mobile wireless devices is increasing and may soon outmum- ber photographs taken with dedicated digital an film based ‘cameras. Thus, there isa growing need (0 improve the expe rionce associated with mobile wireless digital photography by providing a broader range of features that is often associ- ste with dediceted digital and film based cameras BRIEE DESCRIPTION OF THE DRAWINGS. 10003] The present disclosure is illustrated by way of ‘example, and not by way of limitation, in the figures oF the ‘accompanying drawings, in which: {0004} FIG. 1 is a ncework diagram depicting » network system having a client-server architecture configured for ‘exchanging data over a network, according to one embod ment, 10005] FIG. 2 shows a block diagram illustrating one ‘example embodiment of a messaging application [0006] FIG. 3 shows a block diagram illustrating one ‘example embodiment of a photo filter application, [0007] FIG. 4 shows @ block diagram illustrating one ‘example embodiment ofa user-based phot filter publication module [0008] FIG. 4B showsan example ofa prapical user inter face fora user-based photo filler publication module. [0009] "FIG. 4C shows an example of an operation of the truphical user interface of FIG. 4B. [0010] FIG. 4D ioststes an example ofa publication of a user-based photo filter [OLI] FIG. 5 shows a block diagram iustating one ‘example embodiment ofa merehant-based photo filter pub= lication module 10012] FIG. 8 ostrates an example of «common geolo- {0013} FIG. SC itsrates an example of an opertion ofa merehant-based photo filer [0014] FIG. 6A shows diggram illustrating an example of photo filler operating with a eount module, [0015] _ FIG. 68 shows diagram illustrating an example of ‘a photo filter operating witha collection module. [0016] FIG. 7 shows a How diagram illustrating one ‘example embodiment ofan operation ofthe user-based photo ‘ler publication module, [0017] FIG. 8 shows a flow diagram illustrating one ‘example embodiment of an operation of the merehant-based photo filter publication module, [0018] FIG. 9 shows a ow diagram illustrating one ‘example embodiment of an operation of the collection mod tl, [0019] FIG. 10 shows 2 Mow diagram illustrating one ‘example embodiment of an operation ofthe count module. Jul. 14, 2016 [0020] FIG. 11 shows « diagrammatic representation of ‘machine, inthe example form of a computer system, within \hiel a set of instructions may be executed to cause the ‘machine Wo perform aay’ one or more of the methodologies siscussed herein, [021] FIG. 12 is o block diagram illsiating a mobile eviee, according to an example embodiment. DETAILED DESCRIPTION 0022} Although the present disclosure is described with reference (specific example embodiments it will be evident ‘hat various modifications and ehanges may be made to these embodiments without departing from the broader spirit and scope ofthe disclosure, Accordingly, the specification and drawings are to be regarded in an ilusraive rather than 3 0023} The addition of labels, drawings and other artwork {o photographs provides « compelling Way for users t per- sonalize, supplement and enhance these photographs bofare Storage oF publication to a broader audience. An example technology provides usors with a set of photo filters (eg. providing enhancements and augmentaions) that ean be Applied to 8 photograph taken by the set. The set of planta ters (ee, image overlays) may be determined based om a recognition of an object in the photograph that satisies speci tied object criteria associated with a photo filter. In this Way, the photo filers are presente to a user for selection and use ‘with the photograph hased on a recognized content of the photograph. For example, ifthe user takos a photograph and ‘an objet inthe photograph is recognized asthe Empire State Building, photo filters associated with the Empire State Building may’ be provided tothe user for use with the photo- raph. In this example a pictureof the Empire State Building ‘may use a King Kong filter that would place the giant ape on the Empire Stale Building inthe photograph a different per- spectives. Therefore, piture of the south face ofthe Empire Slate Building might see King Kong's back, while a picture from the noth face might soo King Kong's face looking at you. In this example, provision ofthe King Kong filter may Also be constrained by a geo-fence (eg, geographic bound- fry) around the area in New York inchuding the Empire State Building. OF course, futher Empire State Building-hemed phot filters may also be presented to the use. The presen fion ofthe photo filters tothe user may be in response to the user performing a gesture (eg. swipe operation) ona sree of the mobile device. Furthermore, although some example embodiments describe the use of filter in conjunction With Photographs, it should he noted that other example embodi- ats contemplate the use of filters with videos. [024] Third party entities (ee, merchants, estauraats, individual users, ete) may: in one example embodiment, cro fate photo filters for inclusion in the set presented Tor user seletion based on recognition ofan object satisfying eritria specified by the ereator of the photo iter For example, a photograph including an object recognized as a restaurant nay result inthe user being presented with phot filters that overly a menu of the restaurant on the photograph. Or a photograph including an object recognized as a food type ‘may result in the user being presented with phot filters that Jet the user view infomnatione g. calories, fat content, costar ‘ther information associated with the food type. Third pary cenlties may also bid (or tlersse purchase opportunities) to fave # photo filter inclided in a set presented to a user for ion of a particular photograph. Described below US 2016/0203586 AI are various systems and methods for implen described photograph 10025] | More specifically, various examples of photo fter pplication are described. The photo’ filter application includes a photo filter publication module that operates at a server, in some embodiments, and generates photo filters based oa filter data associated with the satisfaction of speci= fied object criteria by objects recognized ins photograph. In ‘ihr embodiments, sme or allo the functionality provided by the photo fier publication module may be resident on client devices, A photo filter may be generated based on. supplied filter data that may include audio andr visual cone ‘eat or visual effects that can be applied to augment the Photograph at a mobile computing device. The photo filter publication module may itself include a user-based photo filter publication module and a merchant-based photo fier publication module. [0026] The photo filter application also includes a photo fer engine that determines that a mobile device has taken & photograph and, based on the photograph including an object that satisfies the objec criteria, provides the photo liter tothe ‘lint device. To this end, the photo filter engine includes an ‘object recognition module configured to find and identify ‘objets in the photograph; and compare each object against the object criteria. The object criteria may inelude associa- tions between an object and a source of image data, for ‘example exhibits in a museum, in which ease the associated photo filer may include images including data asso with a specific exhibit in dee museum, 10027] Using the user-based photo fiter publication mod- ule the photo filter publication application provides Graphi- ‘al User Interface (GUI) fora user to upload iter data for enerating a photo filter and object criteria for comparing t© recognized objects in a photograph. For example, the user ‘may uploada log image forthe creation of «photo filter and specify itera that must be satisfied by an abject recognized inthe photograph inorder forte filter to be made available samobiledeviee. Once the user submits the logo and specifies the object eritera, the photo filter publication mole gener tesa photo filler that inclides the logo ands associated with satisfaction of the specified abject criteria. As such, mobile ‘devices tat have taken a photograph including a recognized ‘object that satisfies the specified object criteria may have socess tothe logo-photo filter 10028] In other examples, if a photograph includes more than specified numberof objects that slisty specified objest

    You might also like